Reverse proxy decodes %2F in upstream paths, and WebSocket connections are counted as permanently in-flight

[barbe1bc]

Environment

• Image: unified-cuda-2026-06-21
• Upstream in question is not an LLM: a ComfyUI container registered as a
  swap-group member so llama-swap arbitrates a single GPU between it and the
  chat models. It serves a browser SPA (aiohttp) rather than an OpenAI API.
• Config shape (trimmed):

models:
  "comfyui":
    cmd: /bin/sh /scripts/comfyui-start.sh     # starts a container, blocks on docker /wait
    cmdStop: /bin/sh /scripts/comfyui-stop.sh
    proxy: "http://comfyui:8188"
    checkEndpoint: "/system_stats"
    ttl: 600
    concurrencyLimit: 128
groups:
  "chat":
    swap: true
    exclusive: false
    members: ["qwen3.6-27b", "...", "comfyui"]

Both issues are independent of the container-based cmd; they are properties of
the proxy layer.

1. The proxy decodes %2F in the upstream path

An encoded slash in the path is decoded before the request is forwarded, so an
upstream route that matches a single path segment stops matching.

Concretely: ComfyUI's route is /userdata/{file}, and aiohttp's {file}
matches one segment. Its frontend requests
/userdata/workflows%2Fmy-workflow.json. Through llama-swap the upstream sees
/userdata/workflows/my-workflow.json and returns 404, so the UI can list
saved workflows but can never open one.

Isolation — same target, four paths:

Request	Result
direct to the upstream container, %2F	200
direct to the upstream container, literal /	404
llama-swap on loopback (no other proxy in front), %2F	404
via a reverse proxy → llama-swap, %2F	404
via a reverse proxy → llama-swap, double-encoded %252F	200
via a reverse proxy straight to the container (llama-swap bypassed), %2F	200

The loopback row rules out anything in front of llama-swap. The %252F row
shows exactly one decode is being applied.

Ask: forward the raw RequestURI rather than a decoded/re-encoded path, so
percent-encoded separators survive. In Go's net/http/httputil, this is the
difference between assigning req.URL.Path and preserving
req.URL.RawPath / req.URL.Opaque when rewriting the director's target.

2. WebSocket connections are counted as permanently in-flight

A long-lived WebSocket to a group member appears to count as an open request for
the lifetime of the connection. Consequences observed:

• The member never drains, so ttl never fires and the model is never unloaded.
• Because the group is swap: true, every request for any other member of
  the same group blocks until timeout and then returns HTTP 200 with an empty
  body. From a client's perspective the whole endpoint is down, with a success
  status code.
• Stopping the upstream out-of-band does not recover it: the browser reconnects
  and llama-swap starts the member again within ~60 s.

Measured on this host:

Observation	Value
Longest single held /ws (one forgotten browser tab)	7h29m17s — logged only when it finally closed
Another, earlier	1h58m
POST /v1/chat/completions for a different group member while held	200, body length 0, after 6m40s
Same request once the WebSocket was gone	200 in 1.3s
Container stopped by hand, then left alone	restarted by llama-swap in ~60 s, WS re-established

Note for anyone debugging this: llama-swap logs a request on completion, so
an open WebSocket is invisible in the logs while it is the thing causing the
problem. The /ws line above only appeared at 7h29m17s once the socket died.
Counting established sockets on the upstream's port is the only reliable check.

Ask: either exclude upgraded connections from the in-flight count used for
drain/ttl decisions, or provide a configurable drain that force-closes
upgraded connections when another member of the group needs to load. A
ttl-exempt-but-evictable state would also solve it.

Why this matters together

The two combine badly for any non-LLM upstream that serves a browser UI: the UI
is unusable for its main purpose (issue 1), and merely leaving its tab open
takes down every other model in the group (issue 2). The workaround deployed
here was to route browser traffic around llama-swap entirely — a separate
reverse-proxy route straight to the container, with llama-swap keeping the
API path and remaining the only component that starts/stops the upstream. That
works, but it means llama-swap no longer sees the UI's traffic at all, so its
ttl had to be disabled for that member.

[mostlygeek]

I have #988 open to fix the url encoding issue.

Websockets are inflight requests and they are active connections so they prevent swapping. Hmm 🤔