Describe the bug
When you set up llama-swap in docker, all the documentation suggests that you can use a config file that calls llama-server plain without any kind of qualification. This doesn't work, trying to start the model will silently fail with a 502, you have to look at the response to see that it's complaining it can't find llama-server in the path.
Upon checking the Docker Deployment Documentation on uhh Devin's auto-generated slop wiki, you can see that the executable is located at /app/llama-server, and using this absolute path in your config makes it work. Either the main documentation should be updated to note this or llama-server should be on the path within the docker container.
Expected behaviour
It would be nice to be able to just call llama-server in my config file, but at the very least I'd like to be informed that I shouldn't. Also, actually showing the error in the UI instead of just doing nothing at all and making me open the network inspector to see it would be cool.
Operating system and version
- OS: (linux)
- GPUs: (irrelevant)
My Configuration
Doesn't work:
models:
GLM-4.5-Air:
cmd: |
llama-server
--port ${PORT}
--model /models/cerebras_GLM-4.5-Air-REAP-82B-A12B-Q5_K_S-00001-of-00002.gguf
--n-gpu-layers 999
--jinja
--ctx-size 131072
--cache-type-k q4_0
--cache-type-v q4_0
--alias ${MODEL_ID}
--flash-attn on
Works fine:
models:
GLM-4.5-Air:
cmd: |
/app/llama-server
--port ${PORT}
--model /models/cerebras_GLM-4.5-Air-REAP-82B-A12B-Q5_K_S-00001-of-00002.gguf
--n-gpu-layers 999
--jinja
--ctx-size 131072
--cache-type-k q4_0
--cache-type-v q4_0
--alias ${MODEL_ID}
--flash-attn on
Describe the bug
When you set up llama-swap in docker, all the documentation suggests that you can use a config file that calls
llama-serverplain without any kind of qualification. This doesn't work, trying to start the model will silently fail with a 502, you have to look at the response to see that it's complaining it can't findllama-serverin the path.Upon checking the Docker Deployment Documentation on uhh Devin's auto-generated slop wiki, you can see that the executable is located at
/app/llama-server, and using this absolute path in your config makes it work. Either the main documentation should be updated to note this or llama-server should be on the path within the docker container.Expected behaviour
It would be nice to be able to just call
llama-serverin my config file, but at the very least I'd like to be informed that I shouldn't. Also, actually showing the error in the UI instead of just doing nothing at all and making me open the network inspector to see it would be cool.Operating system and version
My Configuration
Doesn't work:
Works fine: