[Bug]: A two-minute stall of the shared OpenCode server under tool CPU load fails every OpenCode thread, though the server recovers #23962

Activity

1. nkoynov commented on Oct 23, 2026

   

   nkoynov

   on Oct 23, 2026

   ContributorAuthor

   More actions

   A second case, a different cause, the same failure (2026-10-23 11:41–11:43 UTC, same versions). The tool processes now run at nice 10, and the OpenCode process wasn't starved this time: a plugin in it kept logging every few hundred ms. But OpenCode serialized all its database work behind one long history load of a 58 MB session (anomalyco/opencode#58437), so /api/event went quiet for 77 s:

   • 11:42:02: T3 logs Lost the OpenCode event stream; reconnecting.
   • After the server came back at 11:42:34, each reconcile request took 3–8.5 s, so RECONCILE_TIMEOUT (15 s) ran out on every attempt.
   • 11:43:42: T3 failed every session.
   • About 33 s later T3 stopped the now-idle server, so the turns that were still running inside OpenCode were lost too.

   The server was alive the whole time, so a reconnect that keeps waiting for a live server (or a reconcile timeout that scales with the number of sessions) would have saved both incidents.
2. nkoynov commented on Oct 23, 2026

   

   nkoynov

   on Oct 23, 2026

   ContributorAuthor

   More actions

   What we run locally until T3 handles this, in case the approach helps: a NODE_OPTIONS=--require preload that wraps the server's globalThis.fetch for the OpenCode 2 event subscription only (GET /api/event with the opencode Basic user). No T3 code is patched; constants are as in 3411/3553/main (SILENT_STREAM_TIMEOUT 45 s, RECONNECT_ATTEMPTS 5, RECONNECT_DELAY 2 s, RECONCILE_TIMEOUT 15 s, none configurable).

   What we found while testing it (VM clone on 3553 + OpenCode 2.1.3, plus a small harness running readEvents and the adapter's reconnect loop on Effect 4.0.4):

   • A stall alone doesn't fail sessions. Under SIGSTOP the reconnect GET simply waits, and the first attempt gets through on SIGCONT. Sessions fail when the server comes back slow: each attempt's reconcile (session list, backfill, permissions and forms) runs into the 15 s timeout, five attempts 2 s apart are spent in about 90 s, and every session fails while the server is alive and recovering. That matches both cases above, where reconcile requests took 3 to 8.5 s each after the stall. In the harness, 60 s stopped plus 100 s of 20 s answers fails every session. On the VM, 3 running sessions whose histories hold ~56 MB of inline images each and a 50 s freeze failed all 3 turns: the stream was lost, all 5 reconciles timed out on the backfill, and the sessions failed 100 s after the loss.
   • While the server's port still accepts connections, the preload keeps a silent stream open with SSE comment lines (only between lines, so it never splits a frame), up to 240 s (under undici's 300 s body timeout).
   • Past that, it holds the reconnect's subscription until /api/session/active answers within 2 s, up to 10 minutes from the stall's start, and backs off between attempts (5, 10, 20, 40, 60 s). A failed attempt is recognisable from outside: the adapter starts reading the stream only after reconcile succeeded, so a subscription left unread for 15 s was dropped by its check.
   • A port that refuses connections (process gone) gets the stock path at once, so a killed server still fails its sessions quickly and is replaced.
   • Same VM case with the preload: no reconnect at all, 3/3 turns completed. 120 s and 200 s stops: same. With the keep-open window forced down so the hold path ran, 2 of 3 resumed. The third session's backfill alone takes longer than 15 s even on a server answering in ~1 s, which no amount of waiting fixes.

   Inside T3 the simpler version is probably: while the owned server process is running (OpenCodeServerOwner already has isRunning) and its port accepts connections, keep retrying the reconnect with backoff instead of a fixed 5 × 2 s, and only fail sessions once the process exits or a much longer budget runs out. A spawned server is T3's own child, so "alive" is known without probing.

   _{Investigated and written by Claude Opus 5.5 (OpenCode) for @nkoynov.}
3. nkoynov commented on Nov 6, 2026

   

   nkoynov

   on Nov 6, 2026

   ContributorAuthor

   More actions

   #30201 made this worse for a server that stalls while alive. SUBSCRIPTION_TIMEOUT = "10 seconds" in OpenCode2Client.ts now applies to every /api/event subscription, so the 5 reconnect attempts are used up in about 58 s. Before, a reconnect to a stopped server that still accepted connections just waited and went through when it resumed (stock 3411/3553 reconnected after 70 s and 200 s SIGSTOPs).

   VM check with two running turns and the OpenCode server SIGSTOPped for 360 s: on nightlies 4205 and 4234 both turns failed 288 s into the stall with "Could not reconnect … timed out after 10s (GET …/api/event)". On 4119 the reconnect was held until the server answered at 362 s, and both turns completed. These runs had the local preload from my comment above, which keeps the stream open and holds the reconnect. Going by the code, without it a stall past about 45 s + 58 s now fails every running turn.

   Could the subscription timeout be configurable, or back off and keep retrying while the server's port still accepts connections? Failing fast still makes sense when the port refuses or the process is gone.
4. nkoynov commented on Nov 9, 2026

   

   nkoynov

   on Nov 9, 2026

   ContributorAuthor

   More actions

   A third way to hit the same failAll, this time with a healthy server (nightly 4340, OpenCode 2.2.6, 2026-11-08 21:27 UTC):

   • 21:25:00–21:25:06: one thread started 15 background subagents at once; about 19 OpenCode sessions were running.
   • 21:27:02: OpenCode ended the /api/event response normally (200 after 8.8 h). The server wasn't stalled: its in-process logs kept writing every second and the stream was never silent. In 2.2.6, EventFeed (packages/server/src/event-feed.ts) gives each subscriber a 4,096-event dropping queue and fails a subscriber that falls that far behind (SubscriberOverflowError). That's the most likely trigger here, but neither side logs why the stream ended: the adapter's follow drops the cause before "Lost the OpenCode event stream; reconnecting."
   • 21:27:02–21:28:30: each of the 5 reconnects did session.active and then backfill for every running session, one at a time, paging message.list 50 at a time. With ~19 running sessions that took longer than RECONCILE_TIMEOUT (15 s) on every attempt. Each attempt also started the backfill over from the first session.
   • 21:28:30: "Could not reconnect … TimeoutError: Operation timed out after '15s'", and failAll failed every running turn and subagent on that server. T3 then dropped the server, so all ~19 OpenCode sessions stopped mid-turn.

   The constants and backfill are unchanged on main c4ce7b0. Could the timeout cover each session's backfill instead of the whole reconcile, or could reconcile keep what it has already backfilled across attempts (or read sessions in parallel)? Logging the cause when the stream ends would also help to tell an overflow from a stall.
5. juliusmarminge

   closed this as completedin #25596on Nov 12, 2026