Feature Request: Idle model unload timeout (router mode / config.ini)

[sammcj]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

It would be great if llama-server when run in router mode and/or with a config.ini could unload models that have been idle for a given time.

For example, you could default it to 300 (seconds) and have models that haven't performed any inference in 5 minutes be unloaded to free up resources.

Motivation

Models current sit loaded until the max loaded models are met, using up system resources.

llama-swap provides this as a ttl setting.

Somewhat related:

• Feature Request: Free up VRAM when llama-server not in use #11703
• Power save mode for server --unload-timeout 120 #4598

Possible Implementation

unload-idle-seconds = 300

[some-custom-important-model]
m = /models/custom.gguf
unload-idle-seconds = 3600

llama-server --unload-idle-seconds 300 --models-dir /models

[ggerganov]

I agree this functionality is quite useful. The Metal backend already indirectly supports it thanks to Apple's wired memory collector (#17766) and I find it very convenient that my models are "unloaded" after a configurable amount of time (currently 3 minutes by default).

[ngxson]

This requires #18163 , so will be done after the parent issue is resolved

[ngxson]

@ggerganov not sure if this idea worth exploring, but I've been thinking about having different options of "sleeping" in side libllama: the one option of sleeping can be to unload the model weight (maybe the full llama_model or just a selection of tensors), another option can be to also free KV memory. or, 2 options can be used together.

[ngxson]

I added a "sleeping" feature for both single-model and router modes: #18228

In the future, it can be quite interesting to allow configure the "level" of sleep (as mentioned in the last comment). We can also allow customize unloading only the main model, draft model, multimodal encoder, or any combination of them.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.