Misc. bug: performance regression in `llama-server` (`ggml-vulkan`)

[kpouget]

Name and Version

master branch

Operating systems

Mac

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server --model /path/to/llama3.2 --host 0.0.0.0 --port 11434 --n-gpu-layers 99

Problem description & steps to reproduce

I think #16736 introduced a regression in the performance of llama-server, when backed by ggml-vulkan (but no regression with ggml-metal, which seems strange):

• on 2025-11-02 b6923

  • TTFT 3777ms
  • ITL 41ms

• on 2025-11-03 b6933

  • TTFT 7459ms
  • ITL 62ms

Note that llama-bench isn't affected, pp512=223t/s and tg128=28t/s didn't degrade.

I looked further at the performance of the commits between these two days (tests ran on another system, so the absolute values are different)

b6924..b6927

    - b6924
    - cd5e3b575 # b6925
    - 2f966b8ed # b6926
    - b6927

and they highlight that the performance dropped occurred with cd5e3b5, when #16736 was merged.

First Bad Commit

cd5e3b5

Relevant log output

[ggerganov]

How do you determine TTFT in your tests?

[kpouget]

# First non empty token is the first token
result.first_token_time = tokens[0]['time']

from what I see there, it's the time between the request (start_time) and the first non empty token.

The load tester extracts the stats about all the TTFT, and my test uses the median value for the regression testing
(I can lookup the rest of the TTFT / ITL stats if that can be useful)

[ggerganov]

If you add -kvu to your command does it restore the TTFT?

[kpouget]

yes indeed, with -kvu the TTFT and ITL performance are back to normal

• b6924 is before the regression
• b6927 is after the regression
• both tests ran with the -kvu flag

[ggerganov]

Since this happens with the Vulkan backend and does not happen with the Metal backend, I think it is because the Vulkan FA implementation does not have the optimization for skipping -INF blocks (#16372). @jeffbolznv can you confirm this is the case?

Since by default now the server uses 4 slots with unified cache (#16736) this is likely resulting in the regression. You can use -kvu -np 1 which would result in the old single-slot behavior where this optimization is not necessary.

[jeffbolznv]

Right, ggml-vulkan does not yet skip -INF blocks. Can you clarify what has changed about the shape of the tensors we'll see and what shape of -INF blocks we should be looking for?

Is it possible to reproduce this change in behavior in llama-bench?

[kpouget]

Is it possible to reproduce this change in behavior in llama-bench?

if you're asking if the performance degradation is visible in llama-bench, according to my tests it's not, the perf didn't change

Note that llama-bench isn't affected, pp512=223t/s and tg128=28t/s didn't degrade.

[ggerganov]

Can you clarify what has changed about the shape of the tensors we'll see and what shape of -INF blocks we should be looking for?

When using the unified KV cache, the different sequences are end up in the same buffer. For example, with 4 slots, when we don't use the unified KV cache, the KV cache buffers can look like this:

# . - empty cell
# N - sequence id

00000000...................
11111111111111111111.......
2222.......................
333333333333...............

With unified KV cache, the buffer is single-stream and can look like this:

00000000111111111111111111112222333333333333..............................

So now, if you are processing a new token for seq 2 for example, the mask would look like this:

# . - masked value (i.e. -INF)

............................0000............

I.e. it attends only to the tokens of sequence 2. But in contrast to the non-unified KV cache, they are not necessarily at the beginning of the buffer.

In the Metal backend, I've made an optimization where I run a quick kernel to detect blocks in the KQ mask full of -INF and then during the FA kernel, simply skip them.

Note that while in this case this is close to optimal, it still has some drawbacks. For example, the unified KV cache could end up looking like this in some cases:

0123013012301230123012301230123012301230123012301230...........................

So now, the logic for searching -INF blocks would not help because every block would likely end up having at least one non masked token. Noting this since there might be better approaches than #16372 (e.g. creating a list of indices).

Is it possible to reproduce this change in behavior in llama-bench?

Currently no. It seems we don't have a good way to bench this case. We should figure out something. It might be easier to add an option in llama-batched-bench for that.

[jeffbolznv]

Is it possible to reproduce this with llama-cli? Or only with llama-server?

[ggerganov]

No, llama-cli does not support multiple sequences.

[jeffbolznv]

@kpouget can you share detailed instructions for how to reproduce this? I haven't really used llama-server.

[kpouget]

$ llama-server --model /path/to/llama3.2 --host 0.0.0.0 --port 11434 --n-gpu-layers 99

$ git clone https://github.com/openshift-psap/llm-load-test
$ cd llm-load-test
$ pip install -r ./requirements.txt

config.yaml

output:
  format: "json"
  dir: "./output"
  file: "output.json"
storage:
  type: local
dataset:
  file: "datasets/openorca_large_subset_011.jsonl"
  max_queries: 3000
  min_input_tokens: 0
  min_output_tokens: 0
  max_input_tokens: 2047
  max_output_tokens: 1024
  max_sequence_tokens: 2048
load_options:
  type: constant #Future options: loadgen, stair-step
  concurrency: 1
  duration: 30
plugin: "openai_plugin"
plugin_options:
  streaming: True
  interface: "http"
  model_name: "/path/to/llama3.2" # change this
  host: "http://localhost:11434"
  endpoint: "/v1/chat/completions"
  use_tls: False
extra_metadata:
  replicas: 1

# update plugin_options.model_name to match the llama-server model name:
$ curl -Ssf localhost:11434/models | jq -r .models[0].name

$ python3 load_test.py --config  ./config.yaml

$ cat output/output.json | jq .summary.ttft.median
90.32535552978516

let me know if this works for you

[jeffbolznv]

Thanks, I was able to get this running. I see it report about 74 vs 58 with -kvu, on RTX 5090. These seem orders of magnitude different from what's in your original chart, but similar enough to your last comment, so maybe this is expected.

[kpouget]

the plots in my top post are from a Macbook M4 Pro. Is the RTX 5090 so much faster than that? (I don't know)
ggml-vulkan is only 75%-80% of ggml-metal, but your figures are on a different level