Skip to content

llama-server isn't on path from within the docker image #423

Description

@FrobtheBuilder

Describe the bug
When you set up llama-swap in docker, all the documentation suggests that you can use a config file that calls llama-server plain without any kind of qualification. This doesn't work, trying to start the model will silently fail with a 502, you have to look at the response to see that it's complaining it can't find llama-server in the path.

Upon checking the Docker Deployment Documentation on uhh Devin's auto-generated slop wiki, you can see that the executable is located at /app/llama-server, and using this absolute path in your config makes it work. Either the main documentation should be updated to note this or llama-server should be on the path within the docker container.

Expected behaviour
It would be nice to be able to just call llama-server in my config file, but at the very least I'd like to be informed that I shouldn't. Also, actually showing the error in the UI instead of just doing nothing at all and making me open the network inspector to see it would be cool.

Operating system and version

  • OS: (linux)
  • GPUs: (irrelevant)

My Configuration
Doesn't work:

models:
  GLM-4.5-Air:
    cmd: |
      llama-server
      --port ${PORT}
      --model /models/cerebras_GLM-4.5-Air-REAP-82B-A12B-Q5_K_S-00001-of-00002.gguf
      --n-gpu-layers 999
      --jinja
      --ctx-size 131072
      --cache-type-k q4_0
      --cache-type-v q4_0
      --alias ${MODEL_ID}
      --flash-attn on

Works fine:

models:
  GLM-4.5-Air:
    cmd: |
      /app/llama-server
      --port ${PORT}
      --model /models/cerebras_GLM-4.5-Air-REAP-82B-A12B-Q5_K_S-00001-of-00002.gguf
      --n-gpu-layers 999
      --jinja
      --ctx-size 131072
      --cache-type-k q4_0
      --cache-type-v q4_0
      --alias ${MODEL_ID}
      --flash-attn on

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions