Power save mode for server --unload-timeout 120

[zwilch]

Feature Description

Would it be possible to set a --unload-timeout flag in "server" mode after that llama.cpp unload the model and free the GPU VRAM, so that it saves power. After a new request it will start to load the model automatically again and waiting for "timeout" period and in wich is no new api call then it will unload again.

Motivation

My GPU needs a lot of power when a model is loaded in RAM and it is waiting for new api task.
For example a NVIDIA P40 24GB needs 9W if nothing is loaded to VRAM.
When you use VRAM with some Bytes the power consumption increses to 50W. But still the GPU is not calculating.

RTX 3060 12GB standby power is 6W => 14W with unused loaded Model and "server" waits for api access

Possible Implementation

Example:
server --unload-timeout 120
after 120s with no api task the model will be unloaded to save energy

[peturparkur]

I would support also having an endpoint that allows unloading the model

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[crashr]

I am working on such a feature. If somebody else is, let's talk.

[ApexArray]

I am working on such a feature. If somebody else is, let's talk.

Any luck implementing this? I I'd really like the ability to free memory after computation as well so that it can be used for other tasks in my pipeline. Something similar to Ollama's keep_alive setting would be great.

[crashr]

@ApexArray This wasn't necessary for me anymore since the idle power consumption of Tesla P40 GPU was solved. But I am using a similiar concept to auto shutdowm my LLM server after a specified time of inactivity. I wrote a script that uses tcpdump to monitor the port llama-server listens on. Mabe you can simply modify it to your needs. It is more or less this https://gist.github.com/crashr/fad6a22ddc62d11f952c344bb65aff8c