Runtime CUDA Memory Ballooning to OOM in Mistralrs while Candle-vllm is Stable

[sempervictus]

Describe the bug

Attaching files, sending RAG data, or images to models being run by mistralrs causes the runtime to allocate more and more memory in the GPU(s) until an OOM crash occurs.

This is reproducible with any coding agent, web search MCP, or shell agent which can send files. The problem occurs both on CC890 devices and CC700.

When a model can be run by candle vllm, its memory profile is stable and no additional use occurs when passing data - context window exhaustion is the problem on that side. Mistralrs can truncate but truncation causes the same effect as the OOM - requries a restart of the service.

Latest commit or version

Current master on both, problem has been ongoing over may revisions.

[EricLBuehler]

Hi @sempervictus - thanks. I think this is really the core of the issue, and it can help to narrow things down.

There are a bunch of issues here regarding this, would it be OK if I close them (I'll mark them as a duplicate of this one)? Ultimately I believe that there is some sort of core issue that can be resolved - will take a look.

[sempervictus]

Please! Lets close the stragglers and consolidate

[sempervictus]

@EricLBuehler not quantization... disabled that for gemma3n, asked it to "identify self" via open-webui and its prompt/tool-prep seems to have crashed it immediately:

but no logs (i'm concerned that the leak we saw earlier is still eating the actual output we want for debugging somewhere in the bowels):

2025-07-29T01:02:15.753101Z  INFO mistralrs_server: OpenAI-compatible server listening on http://0.0.0.0:7651.
2025-07-29T01:02:18.102919Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.80, Prefix cache hitrate 0.00%, 0 running, 0 waiting

much simpler client like aichat seems to work fine till you send it a file or significant RAG.

What changes in mistralrs from candle-vllm in terms of GPU memory usage? I'm able to reproduce this on modern (4090) HW so i dont think its the older V100s at fault here (which, when they can run a model on candle-vllm, do so very well and with minimal memory fluctuation at F16 despite base memory usage being significant)

Maybe we need something like nvtop constantly watching GPUs relative to requests to detect faults and "erlang them" so to speak to avoid runaway CUDA threads burning chips?

[sempervictus]

@EricLBuehler so i may have misspoken on the phi4MM piece. When passing it a ~3500 word JSON file, it crashes in a similar-looking way to gemma3n

mistralrs-svc1  | 2025-08-03T05:15:00.319670Z  INFO mistralrs_server: OpenAI-compatible server listening on http://0.0.0.0:7651.
mistralrs-svc1  | 2025-08-03T05:15:02.001722Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.60, Prefix cache hitrate 0.00%, 0 running, 0 waiting
mistralrs-svc1  | 2025-08-03T06:32:52.057398Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 1.80, Prefix cache hitrate 0.00%, 0 running, 0 waiting
mistralrs-svc1  | 2025-08-04T01:11:02.835920Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 2580.40, Prefix cache hitrate 33.33%, 1 running, 0 waiting

no actual OOM reported, client crashed with

$ <request_with_file>
三
Error: Failed to call chat-completions api

Caused by:
    expected value at line 1 column 

[sempervictus]

@EricLBuehler - the Qwen3 MoE issues i'm seeing over in candle-vllm might be a clue to what's going on. The MoE coder model appears to slowly build up CUDA memory usage over time too but at a much much slower rate. When quantizing down to q4k i've so far been unable to make it do the invalid instruction bit at near full memory pressure (there is a separate issue where it seems to think something is q8_1 when nothing should be) but in native FP16 over several hours of letting forge figure out if and how it can port model code from upstream candle, i have managed to get it to a point where the utilization went from 62% to 93% and then started throwing errors.

[sempervictus]

@EricLBuehler - some more data, ironically from qwen3-coder analyzing its own crash data: candle-vllm does significantly increase memory usage at runtime when the context length is sufficiently large (that thing supports 260k, 1m with yarn) but we can see it decreasing memory usage as it approaches capacity across all 4 GPUs: