Repository navigation
fix(server): one runaway agent no longer freezes every agent - #14
Merged
Merged
Conversation
MemoryHigh on app-t3code-agents.slice only throttles. One 8.2 GB process kept the slice between high and max for hours, and every agent in it, including the shared Codex app-server, sat in D state. OOMPolicy=stop then made any kernel kill stop the whole scope, which for Codex ends every thread on the shared app-server. Drop both T3 memory limits and clear any an earlier build left on the slice before the startup probes run. Launch scopes with OOMPolicy=continue, so the kernel kills one process and the rest of its scope keeps running. A live scope that lost a process no longer counts as stopping. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MemoryHighonapp-t3code-agents.sliceonly throttles, and nothing watches it. On forge, one 8.2 GB python process kept the slice between high and max for 6 h 20 m. Every agent sat in D state in__mem_cgroup_handle_over_high, the shared Codex app-server froze, and Stop and steer hung (memory.events: high 6.6M, max 0, oom_kill 0). Separately,OOMPolicy=stopturns any kernel kill into a stop of the whole scope, which for Codex ends every thread on the shared app-server (pingdotgg#16502).Summary
The agents slice has no T3 memory limits. The host or container limit is the hard cap, and
oom_score_adj800 for agents vs 200 for the server makes the kernel pick an agent.stopscopes for the server's lifetime.continueapplies to every scope, not just shared servers (compare fix(server): keep the shared Codex app-server scope alive after a tool OOM kill pingdotgg/t3code#18329 and fix(opencode): a tool killed for memory no longer stops every OpenCode thread pingdotgg/t3code#19367). The kernel kills one process and the rest of the scope keeps running.classifyScopeno longer treats a live scope withoom_kill > 0as stopping.oomPolicywas alwaystrue. It now reports whethercontinuewas applied.Evidence
Before: the new assertions fail against the old
agentScope.ts:After:
vp test run apps/server/src/process/agentScope.test.ts: 7 passed. Server and shared typecheck, lint and format are clean.Before: real systemd 255 on forge, a throwaway scope outside the agents slice:
After:
Merge Danger
Door: two-way
Reverting restores the old limits the next time a server launches its first agent.
Blast Radius: Linux
Only Linux hosts with a systemd user manager run agent scopes.
OOMPolicyon scopes, so it still stops the whole scope on an OOM kill, as before.Result=oom-kill, so the "Killed: out of memory." label stops firing. That path (AgentScope.oomKilled/clear, theProviderEventIngestorrelabel, the per-thread scope tracking) is left for a separate deletion change.After deploying, check
systemctl --user show app-t3code-agents.slice -p MemoryHigh -p MemoryMaxreadsinfinityonce the new build has launched an agent.Model and harness: Claude Opus 5.5 in Claude Code via T3 Code.
🤖 Generated with Claude Code