Repository navigation
fix(api): honor a systemd CPUQuota and bound ONNX Runtime threads (#5725) - #5779
Merged
Merged
Conversation
…NNX Runtime sessions (#5725) The native thread cap read only the cgroup mount root, so a systemd unit's CPUQuota= (set on /system.slice/<unit>) was never seen and every pool was sized to the host. ONNX Runtime also ignores OMP_NUM_THREADS, so FlashRank, ONNX embeddings and the onnx reranker (by default) started one spinning thread per physical core even when the quota was found.
…hs, cover the summary - bound_onnx_session takes ort.SessionOptions (TYPE_CHECKING import) instead of Any - _cgroup_cpu_quota skips a /proc/self/cgroup path containing '..' (cgroup outside the namespace root) instead of joining it onto the mount - FlashRank comment says why the session is always rebuilt - onnx reranker's threads<0 error no longer says "0 = ONNX Runtime default" - tests for native_thread_summary, bad OMP_NUM_THREADS, and the '..' path
This was referenced Oct 20, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5725.
Problem
On a many-core host where the service is held to a few CPUs, local reranking and embedding start one busy thread per core, use up the quota spinning, get throttled every scheduler period, and take minutes instead of seconds. Two causes:
CPUQuota=sits on the process's own cgroup (/system.slice/<unit>.service). The root has no limit, so the cap fell back to the host core count (min(16, cores)).OMP_NUM_THREADS. FlashRank and ONNX embeddings never set a thread count, and neither does theonnxreranker unlessRERANKER_ONNX_THREADSis set. Each session sized itself to all physical cores, with spinning on. This happens even in normal containers where the quota is detected.Fix
_cgroup_cpu_quota()reads/proc/self/cgroup, walks from the process's own cgroup up to the mount (v2cpu.max, v1cpucontroller cfs files), and keeps the tightest quota. The mount root stays as a fallback, so containers behave as before.bound_onnx_session()is used by all three places that build an ONNX session. It setsintra_op_num_threadsto the native cap (OMP_NUM_THREADS, so an operator-set value still wins),inter_op_num_threads=1, and turns spinning off. FlashRank now always rebuilds its session, not only when the memory arena is off.Behaviour changes
min(16, available CPUs)threads by default instead of all physical cores.RERANKER_ONNX_THREADS=0now means "the native cap", not "all cores".onnxreranker already had it off).Evidence
Docker with
--cgroupns=host(the process sits in a child cgroup with the limit set there, like a systemd unit) on a 6-core VM; 300 runs of a 1024×1024 MatMul, 3 runs each:--cpus=1, host cgroupns--cpus=2, host cgroupns--cpus=2, private cgroupnsNew tests cover: the systemd unit quota, the tightest parent winning, the container root, cgroup v1, no limit, the ONNX session settings, and FlashRank's session rebuild.