Changing context_size in YAML Configuration Editor requires restarting

[loopy321]

LocalAI version:

v.3.12.1

Environment, CPU architecture, OS, and Version:

docker

Describe the bug

When changing the model .yaml file via the frontend to change the context_size: 16384, after clicking the "Update" button, the models appear to reload but the change in the yaml is not used. If I simply restart the LocalAI docker container, the change is used. I believe this used to work as expected in a prior version of LocalAI.

To Reproduce

Make a change in the model's context_size in yaml via the frontend's "YAML configuration editor" and click save. Note the change is not implemented.
Expected behavior

changed configuration.

Logs

Config:

backend: llama-cpp
context_size: 16384
f16: true
mmap: true
mmproj: mmproj/mmproj-Qwen3-VL-2B-Instruct-F16.gguf
name: qwen3-vl-2b-instruct
parameters:
    model: Qwen3-VL-2B-Instruct-Q4_K_M.gguf
...

Log:
Feb 24 14:02:53 DEBUG GRPC stderr id="qwen3-vl-2b-instruct-127.0.0.1:34731" line="srv send_error: task id = 321, error: request (8899 tokens) exceeds the available context size (8192 tokens), try increasing it" caller={caller.file="/build/pkg/model/process.go" caller.L=153 }

Additional context

[xXMrNidaXx]

Good catch! The YAML update likely writes the file but doesn't trigger a model reload with the new context size.

The issue: context_size is set when the model is first loaded into memory. Changing the YAML doesn't reinitialize the already-loaded llama.cpp backend.

Likely fix location:
The config update endpoint should call something like:

func (api *API) UpdateModelConfig(modelName string, config *Config) error {
    // Write config to YAML
    if err := config.Save(); err != nil {
        return err
    }
    
    // Force model unload + reload with new config
    if err := api.modelLoader.UnloadModel(modelName); err != nil {
        log.Warnf("unload error: %v", err)
    }
    
    // Next request will reload with new context_size
    return nil
}

Workaround:

# After updating YAML via frontend, call:
curl -X POST http://localhost:8080/models/apply

# Or delete model from memory:
curl -X DELETE http://localhost:8080/models/{model_name}

The model will reload on next request with the new context_size.

---

Nice find! 🎯 If you run production LLM inference:

https://www.revolutionai.io/find-work

Production AI projects • Flexible hours • Great rates 💼