Skip to content

fix(api): honor a systemd CPUQuota and bound ONNX Runtime threads (#5725) - #5779

Merged
nicoloboschi merged 2 commits into
mainfrom
triage/5725
Oct 19, 2026
Merged

nicoloboschi merged 2 commits into
mainfrom
triage/5725

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Fixes #5725.

Problem

On a many-core host where the service is held to a few CPUs, local reranking and embedding start one busy thread per core, use up the quota spinning, get throttled every scheduler period, and take minutes instead of seconds. Two causes:

  1. The CPU quota was only read from the cgroup mount root. That works in a container with its own cgroup namespace, but a systemd unit's CPUQuota= sits on the process's own cgroup (/system.slice/<unit>.service). The root has no limit, so the cap fell back to the host core count (min(16, cores)).
  2. ONNX Runtime ignores OMP_NUM_THREADS. FlashRank and ONNX embeddings never set a thread count, and neither does the onnx reranker unless RERANKER_ONNX_THREADS is set. Each session sized itself to all physical cores, with spinning on. This happens even in normal containers where the quota is detected.

Fix

  • _cgroup_cpu_quota() reads /proc/self/cgroup, walks from the process's own cgroup up to the mount (v2 cpu.max, v1 cpu controller cfs files), and keeps the tightest quota. The mount root stays as a fallback, so containers behave as before.
  • New bound_onnx_session() is used by all three places that build an ONNX session. It sets intra_op_num_threads to the native cap (OMP_NUM_THREADS, so an operator-set value still wins), inter_op_num_threads=1, and turns spinning off. FlashRank now always rebuilds its session, not only when the memory arena is off.
  • At startup the log prints the per-pool thread count, the available CPUs and the quota found.

Behaviour changes

  • ONNX sessions use min(16, available CPUs) threads by default instead of all physical cores. RERANKER_ONNX_THREADS=0 now means "the native cap", not "all cores".
  • Spinning is now off for FlashRank and ONNX embeddings (the onnx reranker already had it off).

Evidence

Docker with --cgroupns=host (the process sits in a child cgroup with the limit set there, like a systemd unit) on a 6-core VM; 300 runs of a 1024×1024 MatMul, 3 runs each:

Setup main this PR
--cpus=1, host cgroupns quota not found, cap 6, 5 ORT threads, 2.1–2.9 s quota 1, cap 1, 0 extra, 0.54–0.76 s
--cpus=2, host cgroupns quota not found, cap 6, 5 ORT threads, 0.79–0.96 s quota 2, cap 2, 1 extra, ~0.49 s
--cpus=2, private cgroupns quota 2, but 5 ORT threads, 1.29 s cap 2, 1 extra, 0.62 s

New tests cover: the systemd unit quota, the tightest parent winning, the container root, cgroup v1, no limit, the ONNX session settings, and FlashRank's session rebuild.

…NNX Runtime sessions (#5725)

The native thread cap read only the cgroup mount root, so a systemd unit's
CPUQuota= (set on /system.slice/<unit>) was never seen and every pool was
sized to the host. ONNX Runtime also ignores OMP_NUM_THREADS, so FlashRank,
ONNX embeddings and the onnx reranker (by default) started one spinning
thread per physical core even when the quota was found.
…hs, cover the summary

- bound_onnx_session takes ort.SessionOptions (TYPE_CHECKING import) instead of Any
- _cgroup_cpu_quota skips a /proc/self/cgroup path containing '..' (cgroup outside
  the namespace root) instead of joining it onto the mount
- FlashRank comment says why the session is always rebuilt
- onnx reranker's threads<0 error no longer says "0 = ONNX Runtime default"
- tests for native_thread_summary, bad OMP_NUM_THREADS, and the '..' path
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native thread limits ignore a systemd unit's CPUQuota, and ONNX Runtime sessions never get a thread count (0.10.2)

1 participant