Misc. bug: Maximum Context Size

[AlphaMo99]

Name and Version

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX 2000 Ada Generation Laptop GPU, compute capability 8.9, VMM: yes
version: 7539 (83b3b1c)
built with GNU 15.2.0 for Linux x86_64

Looks like when we specify the context size to zero (eg: -c0 ), it's now defaulting to 4096 instead of the maximum size of the model.

Operating systems

No response

Which llama.cpp modules do you know to be affected?

No response

Command line

llama-server --port 10004 --host 127.0.0.1  --api-key xxxxx --jinja -fa on -hf mradermacher/Ling-lite-GGUF --cache-type-k q8_0 --cache-type-v q8_0 -c 0

Problem description & steps to reproduce

llama_context: n_ctx = 4096
llama_context: n_ctx_seq = 4096
llama_context: n_batch = 2048
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = enabled
llama_context: kv_unified = true
llama_context: freq_base = 600000.0
llama_context: freq_scale = 1
llama_context: n_ctx_seq (4096) < n_ctx_train (16384) -- the full capacity of the model will not be utilized

It is taking 4096 context size instead of 16384.

First Bad Commit

No response

Relevant log output

[JohannesGaessler]

Should be the same problem as in #18266 but for the context size instead of -ngl. Because 0 is the default for the context size it is not being recognized as having been set by the user and therefore changed in llama_params_fit. For now it can be fixed by setting --fit off or by setting --fit-ctx to the minimum that should be left when trying to reduce memory usage.

In any case, please post the logs you get without the above options but with --verbose. It may be that there is another issue.

[AlphaMo99]

Dear,

Please find the log, the context size of the model is 16384, the option -c0 is setup, but llama is using the default value 4096 (It was working well maybe two or three weeks ago)

log.txt

[grapevine-AI]

Let me explain carefully.
-c 0 no longer refers to the maximum model size as you mentioned. Instead, it now means that the size is automatically determined within the range from 4096 to the maximum model size, based on available VRAM.
(See #16653)
Aren't you using a model that's too large for your VRAM size? In this situation, llama.cpp likely determined it couldn't handle long contexts and automatically set the minimum size of 4096.

If you don't mind the KV cache overflowing from VRAM, you can disable this feature using --fit off.

[JohannesGaessler]

Only one small amendment:

-c 0 no longer refers to the maximum model size as you mentioned.

Technically this still means the maximum context size. The problem is just that with -fit on this is not being differentiated between the default value and this exact value set by the user (in which case it should not be changed).

[AlphaMo99]

Ok sad, I liked the previous way,
the --help should be updated then :

-c, --ctx-size N size of the prompt context (default: 0, 0 = loaded from model)

[JohannesGaessler]

The previous functionality will be restored, it is a bug that right now -c 0 is being changed with -fit on.

[65a]

Seeing something similar and a bit weird with the C API.

If I create default params structs using llama_context_default_params() and friends, then call fit on them with a "minimum" size of 16384, I get params back from fit with context set to 512 and a return code of zero from fit.

Possible I'm doing something weird, just switched over to try using fit.

[JohannesGaessler]

The way llama_params_fit is currently implemented is inconsistent, you need to set the context size explicitly to 0 which is the default of the common API but not of llama_context_default_params.

[65a]

@JohannesGaessler for some reason this still doesn't seem to help. I'm calling llama_context_default_params(), then setting n_ctx to 0 in the result. If I then call params_fit and pass it that context, I get back a context which is still set zero n_ctx. For now I just set the desired context as before (known to fit), and params_fit doesn't decrease it. Layer offloading and everything seems to work fine. This works, not sure what I did wrong the first time.

This works in some cases but not others (glm worked, gemma did not), and I'm not 100% sure where and why. GLM fits only with reduced context, gemma fits with full context, if that helps. It works best if I preset n_ctx to n_ctx_train, which is hard to get if I haven't actually loaded the model yet. I can set it to say 1M, but this has weird performance side effects even after the value is trimmed down.

I do want to say thanks, this is awesome and it makes things much easier.