Skip to content

fix(core): clean up orphaned MCP processes after a crash - #59398

Closed
rekram1-node wants to merge 2 commits into
devfrom
mcp-orphans
Closed

rekram1-node wants to merge 2 commits into
devfrom
mcp-orphans

Conversation

@rekram1-node

@rekram1-node rekram1-node commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem

If the opencode server crashes or gets SIGKILLed, local MCP servers that don't exit on stdin EOF (like npx wrappers) keep running forever, reparented to PID 1. They pile up across restarts. Part of #50363.

Normal shutdown already kills the MCP process group, but nothing does that when the server dies without running its cleanup.

Change

I start each local MCP server through a small sh guard in packages/core/src/mcp/stdio.ts. The server gives the guard an extra pipe (fd 3) and never writes to it. The kernel closes that pipe however the server exits, so a watcher in the guard sees EOF and SIGKILLs its own process group.

opencode server ──fd 3 pipe──┐
                             ▼
sh guard (group leader)
├── watcher: read fd 3 → EOF → kill -KILL 0
└── MCP server (stdin/stdout passed through)
  • No state files or startup sweep, nothing shared between servers, and cleanup happens right away.
  • kill 0 hits the watcher's own group, which exists as long as the watcher does, so a reused PID can't be hit.
  • When the MCP server exits on its own, the guard stops the watcher and exits with the server's status, so normal stops work as before.
  • Only applies to servers spawned on this host on macOS and Linux. Workspace-placed servers run elsewhere, so they aren't wrapped.

Out of scope

  • Windows. There are no process groups to signal there; doing this properly needs a job object with kill-on-close, which needs native code.
  • Shell tool jobs and other process groups. The same trick would work, but wrapping every spawn changes PIDs and exit statuses for many more callers.

Verification

  • Repro: serve with a stdio MCP server that ignores stdin EOF, then kill -9 the server. Before: the MCP server stays alive with PPID 1. After: it's gone immediately.
  • Disconnect, reconnect and SIGTERM shutdown still leave nothing behind.
  • bun run script/test.ts test/mcp.test.ts and bun typecheck in packages/core, bun run check from the root.

Visual

<svg xmlns="http://www.w3.org/2000/svg" width="240" height="72" viewBox="0 0 240 72" fill="none" stroke="currentColor" stroke-width="2">
<style>
.s{animation:s 4s infinite}.l{animation:l 4s infinite}.c{animation:c 4s infinite}.x{opacity:0;animation:x 4s infinite}
@keyframes s{0%,30%{opacity:1}38%,85%{opacity:.25}95%,100%{opacity:1}}
@keyframes x{0%,30%{opacity:0}38%,85%{opacity:1}95%,100%{opacity:0}}
@keyframes l{0%,32%{opacity:.5}40%,85%{opacity:0}95%,100%{opacity:.5}}
@keyframes c{0%,42%{opacity:1;transform:none}55%,85%{opacity:0;transform:translateY(6px)}95%,100%{opacity:1;transform:none}}
@media (prefers-reduced-motion:reduce){.s,.l,.c,.x{animation:none}}
</style>
<rect class="s" x="16" y="20" width="64" height="32" rx="6"/>
<path class="s" d="M32 36h32" opacity=".6"/>
<path class="x" d="M38 26l20 20M58 26L38 46" stroke-width="3"/>
<path class="l" d="M80 36h48" stroke-dasharray="4 4"/>
<g class="c"><rect x="128" y="20" width="44" height="32" rx="6"/><rect x="184" y="20" width="44" height="32" rx="6" opacity=".6"/><path d="M172 36h12" opacity=".6"/></g>
</svg>

Added by the opencode bot (AI-generated).

Related: #50363

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix works. On the base, kill -9 of serve left a stdio MCP server that ignores stdin EOF, and its child, running with PPID 1. With this PR they're all gone within 3s. Disconnect, reconnect and SIGTERM shutdown still leave nothing behind. The core typecheck passes, and the MCP tests match the base: the one failure there also fails on the base.

Public API: unchanged.

Worth knowing: a nonexistent command now fails as MCP server process exited with code 127: …: not found instead of the spawner's NotFound error. It's still readable, since stderr is included.

The SIGKILL test in process-guard.test.ts fails here on every run. The guard does its job, but the test counts zombies as alive (see inline).

Related: #50363

Comment thread packages/util/src/process-guard.test.ts Outdated

// SIGTERM ends the descendant at once; the leader ignores it until the SIGKILL two seconds later.
const deadline = Date.now() + 10_000
while (pids.some(alive) && Date.now() < deadline) await Bun.sleep(50)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fails every run in our sandbox, even though the guard works. The killed leader ends up a zombie (STAT Z, PPID 1) because PID 1 there doesn't reap orphaned processes, and process.kill(pid, 0) succeeds for zombies. Containers without an init process behave the same way. Could alive treat zombies as dead (e.g. check the state in /proc/<pid>/stat, or ps -o stat=)? Then the test checks that the processes stopped, not whether PID 1 reaped them. The real 2s/10s waits also make it slow; a test with only the descendant (no TERM-ignoring leader) would avoid the 2s escalation.

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still fixes the crash case. I reran the serve + kill -9 repro with a stdio MCP server that ignores stdin EOF. On the base it stays alive with PPID 1; with this PR it and its child are gone right away. Disconnect, reconnect and SIGTERM shutdown still leave nothing behind. The core typecheck passes, and the MCP tests match the base.

The earlier point about the flaky SIGKILL test is resolved: that test and process-guard.ts were removed, and the guard is now inline in stdio.ts.

Public API: unchanged.

No new concerns.

@rekram1-node

Copy link
Copy Markdown
Contributor Author

Closing this in favor of a smaller approach: track the process groups the spawner starts and kill them in an exit hook, instead of wrapping every spawn in sh. I'll open that as a separate PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant