Skip to content

fix(server): one runaway agent no longer freezes every agent - #14

Merged
asonawalla merged 1 commit into
mainfrom
remove-agent-memory-limits
Oct 17, 2026
Merged

asonawalla merged 1 commit into
mainfrom
remove-agent-memory-limits

Conversation

@asonawalla

Copy link
Copy Markdown
Owner

MemoryHigh on app-t3code-agents.slice only throttles, and nothing watches it. On forge, one 8.2 GB python process kept the slice between high and max for 6 h 20 m. Every agent sat in D state in __mem_cgroup_handle_over_high, the shared Codex app-server froze, and Stop and steer hung (memory.events: high 6.6M, max 0, oom_kill 0). Separately, OOMPolicy=stop turns any kernel kill into a stop of the whole scope, which for Codex ends every thread on the shared app-server (pingdotgg#16502).

Summary

The agents slice has no T3 memory limits. The host or container limit is the hard cap, and oom_score_adj 800 for agents vs 200 for the server makes the kernel pick an agent.

 first agent or terminal launch
   stop agents a previous server left
+  set-property --runtime app-t3code-agents.slice MemoryHigh=infinity MemoryMax=infinity
   probe systemd-run --scope
-    --property=OOMPolicy=stop
+    --property=OOMPolicy=continue
-  set-property --runtime app-t3code-agents.slice MemoryHigh=max(RAM-6G, RAM/2) MemoryMax=max(RAM-4G, 0.75*RAM)

Evidence

  • Before: the new assertions fail against the old agentScope.ts:

    × does not wait on a live agent or a scope that ended for another reason
        expected 'stopping' to be 'running'
    × launches scopes that outlive an OOM kill on a slice without memory limits
        expected [ '-c', …(13) ] to include '--property=OOMPolicy=continue'
    

    After: vp test run apps/server/src/process/agentScope.test.ts: 7 passed. Server and shared typecheck, lint and format are clean.

  • Before: real systemd 255 on forge, a throwaway scope outside the agents slice:

    $ systemd-run --user --scope -p MemoryMax=256M -p OOMPolicy=stop -- sh -c 'python3 -c "b=b\"x\"*(1<<30)"; echo exit=$?; sleep 1'
    Terminated                     # systemd stopped the whole scope
    Result=oom-kill
    

    After:

    $ systemd-run --user --scope -p MemoryMax=256M -p OOMPolicy=continue -- sh -c '…same…'
    Killed
    exit=137                       # only python died; the shell kept running
    Result=success                 # scope finished normally and was unloaded
    

Merge Danger

Door: two-way

Reverting restores the old limits the next time a server launches its first agent.

Blast Radius: Linux

Only Linux hosts with a systemd user manager run agent scopes.

  • Nothing in T3 caps agent memory now. On a host with no container or cgroup limit, the global OOM killer is the backstop, and it still picks the agent first.
  • systemd 252 (Debian 12) can't set OOMPolicy on scopes, so it still stops the whole scope on an OOM kill, as before.
  • On systemd 253 and later, systemd no longer records Result=oom-kill, so the "Killed: out of memory." label stops firing. That path (AgentScope.oomKilled/clear, the ProviderEventIngestor relabel, the per-thread scope tracking) is left for a separate deletion change.
  • An older build that launches its first agent after this one re-applies its limits until this server restarts.

After deploying, check systemctl --user show app-t3code-agents.slice -p MemoryHigh -p MemoryMax reads infinity once the new build has launched an agent.

Model and harness: Claude Opus 5.5 in Claude Code via T3 Code.

🤖 Generated with Claude Code

MemoryHigh on app-t3code-agents.slice only throttles. One 8.2 GB process
kept the slice between high and max for hours, and every agent in it,
including the shared Codex app-server, sat in D state. OOMPolicy=stop then
made any kernel kill stop the whole scope, which for Codex ends every
thread on the shared app-server.

Drop both T3 memory limits and clear any an earlier build left on the
slice before the startup probes run. Launch scopes with
OOMPolicy=continue, so the kernel kills one process and the rest of its
scope keeps running. A live scope that lost a process no longer counts
as stopping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:M labels Oct 17, 2026
@asonawalla
asonawalla merged commit bb7e7be into main Oct 17, 2026
7 of 22 checks passed
@asonawalla
asonawalla deleted the remove-agent-memory-limits branch October 17, 2026 22:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant