Misc. bug: Qwen3-Next token generation performance regression (CPU-only)

[nifgraup]

Name and Version

.

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

No response

Command line

Problem description & steps to reproduce

Regression point a5251ca (#17996).

before:

./build/bin/llama-bench --threads 12 --device none --model qwen3_next_80b_a3b_instruct-iq4_nl.gguf
| model                          |       size |     params | backend    | ngl | dev          |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------ | --------------: | -------------------: |
| qwen3next 80B.A3B IQ4_NL - 4.5 bpw |  41.98 GiB |    79.67 B | CUDA       |  99 | none         |           pp512 |         64.40 ± 3.75 |
| qwen3next 80B.A3B IQ4_NL - 4.5 bpw |  41.98 GiB |    79.67 B | CUDA       |  99 | none         |           tg128 |          6.65 ± 0.01 |

build: fb644247d (7431)

after:

./build/bin/llama-bench --threads 12 --device none --model qwen3_next_80b_a3b_instruct-iq4_nl.gguf
| model                          |       size |     params | backend    | ngl | dev          |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------ | --------------: | -------------------: |
| qwen3next 80B.A3B IQ4_NL - 4.5 bpw |  41.98 GiB |    79.67 B | CUDA       |  99 | none         |           pp512 |         67.14 ± 0.92 |
| qwen3next 80B.A3B IQ4_NL - 4.5 bpw |  41.98 GiB |    79.67 B | CUDA       |  99 | none         |           tg128 |          5.02 ± 0.00 |

build: a5251ca11 (7432)

AMD Ryzen 9 5900X, RAM clocked at 2133 MHz

First Bad Commit

No response

Relevant log output

[kkontny]

Refactor in #17996 introduced a lot of GGML_OP_SUM_ROWS ops with large sizes, GGML_OP_SUM_ROWS is executed on 1 thread only in GGML.

[nifgraup]

tg128 speed with --threads 12 is 75% of pre-regression speed as shown above. With --threads 1 it is 92%.

[beginor]

I have tried qwen3-next-80b-a3b-thinking on mac m1 max with both mlx_lm (version: 0.29.1) and llama.cpp (version: 7445), the results are:

With mlx engine, the output speed is about 40 tok/sec, the command I use is:

mlx_lm.server --port ${PORT} --log-level INFO
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0
  --max-tokens 16384
  --model ${ROOT_PATH}/qwen/qwen3-next-80b-a3b-thinking-mlx-4bit

But with llama.cpp engine, the output speed is only about 18 tok/sec, the command I use is:

llama-server --port ${PORT} --n-gpu-layers 999 --flash-attn on --ctx-size 32768
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --repeat-penalty 1.0
  --model ${ROOT_PATH}/qwen/qwen3-next-80b-a3b/qwen3-next-80b-a3b-thinking-ud-q4_k_xl.gguf
  --jinja

So, I think the generation speed of llama.cpp should still can be optimized.

I have tried qwen3-30b-a3b models before with mlx and llama.cpp, both can get about 40 tok/sec output.

[nifgraup]

tg speed is regained with the work in #18792 :

| model                          |       size |     params | backend    | ngl | dev          |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------ | --------------: | -------------------: |
| qwen3next 80B.A3B IQ4_NL - 4.5 bpw |  42.00 GiB |    79.67 B | CUDA       |  99 | none         |           pp512 |         74.54 ± 0.12 |
| qwen3next 80B.A3B IQ4_NL - 4.5 bpw |  42.00 GiB |    79.67 B | CUDA       |  99 | none         |           tg128 |          6.54 ± 0.00 |