Skip to content

OpenCode: a live but frozen server still fails every thread, because the reconnect budget ends before reconcile #37396

Closed as not planned

Description

@satyalyadav

What happened

Two OpenCode threads on one T3 server failed in the same second (2026-11-15 21:39:34 UTC) with The OpenCode event stream was lost and could not reconnect. The T3-owned opencode serve process stayed alive and answered requests again about 90 s later. This is the same class as #23962 and recurs on a build that contains #25596.

Diagnosis

T3 drops /api/event after 45 s of silence, then reconnects 5 times, 2 s apart. Each attempt borrows the connection and subscribes before reconciling, and the subscribe has a 10 s timeout. All five attempts died in the subscribe step, so reconcile() never ran and failAll failed every OpenCode thread on the shared session. #25596's unbounded retry for a live owned process only covers reconcile(), so it never applied.

Steps to reproduce

Deterministic, no memory pressure needed: freeze the owned opencode serve with SIGSTOP for longer than about 100 s while a turn is active.

  1. Start an isolated T3 server so your normal one is untouched:
    t3 serve --mode web --port <port> --host 127.0.0.1 --no-browser --base-dir <base-dir>
    
  2. A fresh base dir has the OpenCode provider disabled. Enable it in <base-dir>/userdata/settings.json:
    {"providerInstances":{"opencode":{"driver":"opencode","enabled":true,"config":{"binaryPath":"opencode"}}}}
  3. Add a project and start a thread whose turn stays active:
    t3 project add <workspace> --base-dir <base-dir>
    t3 thread start <workspace> "Run this exact command and wait for it to finish: sleep 600" --base-dir <base-dir> --provider opencode --model opencode-go/deepseek-v4.1-flash
    
  4. Once the sleep 600 tool call is running, stop the opencode serve that this T3 server started (<opencode-port> is the port T3 picked for it, visible in ps), wait 120 s, then resume it:
    pid=$(pgrep -f 'opencode serve --hostname=127.0.0.1 --port=<opencode-port>')
    kill -STOP "$pid"
    for i in $(seq 12); do sleep 10; awk '{print $3}' "/proc/$pid/stat"; done   # T every time
    kill -CONT "$pid"
    

Expected: T3 waits for the live owned server and the turn continues after SIGCONT.
Actual: after 45 s of silence and five 10 s subscribe timeouts, the run fails with transport_error / The OpenCode event stream was lost and could not reconnect. while the process is still in state T. The full timeline from this repro is in the comment below.

How it happened in the real incident

The agents cgroup was at its limit. T3 caps app-t3code-agents.slice at 3.35 GiB soft / 5.03 GiB hard on this 6.7 GiB WSL2 VM, and that covers both provider servers, their tool children and the preview browser. A build with a 3 GB Node heap ran in one OpenCode thread. Slice counters: memory.events high 103941, memory.peak 3.68 GB, swap.peak 2.1 GB, swap.events fail 28018492, oom_kill 0. Nothing was killed; the server stayed throttled longer than T3's window of up to 58 s of reconnects. The T3 server outside the slice stayed responsive.

Version

0.1.10-nightly.20261115.4749 (commit 295a0a7, includes #25596). OpenCode 2.3.0.

Environment

WSL2, 6.7 GiB RAM; T3 desktop app on the Windows host, server inside WSL.

Evidence

21:36:38.253  last OpenCode reply before the stall
21:37:56      last /api/event frame (inferred from the 45 s watchdog)
21:38:41.413  reconnect 1   7.2 s  no response headers
21:38:50.641  reconnect 2  10.0 s
21:39:02.646  reconnect 3  10.0 s
21:39:14.652  reconnect 4   7.2 s
21:39:23.869  reconnect 5  10.0 s
21:39:33.88   failAll; provider session status=error
21:41:02.278  next successful OpenCode reply (GET /api/model, 61.8 s)

OpenCode logged eight /api/event responses closing at 21:41:02 (the five reconnects plus three probes), minutes after T3 gave up.

Related issues

#23962 was closed by #25596; its 2026-11-14 comment reports this remaining subscribe gap. #30201 added the 10 s subscribe timeout that now bounds each reconnect. #25751 is the defect variant that leaves the session ready, not this case. #26171: the shared Codex app-server sits in the same agents slice.

Fix applied or workaround

Server restart cleared it. Local: raise the WSL memory limit, run large builds outside the agents slice, lower the Node heap.

Filed by

DeepSeek V4.1 Flash (OpenCode) via t3 triage, reviewed by Claude Opus 5.5 (Kiro) in T3 Code.

Activity

  1. juliusmarminge commented on Nov 15, 2026

    @juliusmarminge
    Member

    I tried to reproduce this on the current build and could not.

    Commit 69f8f59 (origin/main). I did not run the app against a frozen OpenCode server. Instead I wrote an adapter test in apps/server/src/orchestration-v2/Adapters/OpenCode2AdapterV2.test.ts, 'recovers an owned server after more than five event subscriptions time out'. It runs the real OpenCode2 V2 adapter against the replay server with an owned (live) process. The first /api/event subscription opens the session, the stream then ends, and the next six subscriptions never return headers. The test advances the TestClock 12 s per attempt (the 10 s SUBSCRIPTION_TIMEOUT plus the 2 s RECONNECT_DELAY), then lets the following subscription and reconcile succeed. On main the turn ends 'failed': after 5 reconnect attempts, failAll ran with 'The OpenCode event stream was lost and could not reconnect.', and reconcile never ran. With a temporary unbounded reconnect retry for owned servers the same test passed ('completed'); I reverted that and pushed no fix. The rest of that file and the OpenCode integration tests (190) still pass with the testkit change.

    Closing as not planned. If it still happens for you, reply with the exact steps and the version you are on and it will be looked at again.

  2. satyalyadav commented on Nov 16, 2026

    @satyalyadav
    ContributorAuthor

    Reproduced on 0.1.10-nightly.20261115.4749 (commit 295a0a7) with a plain SIGSTOP and no memory pressure.

    Setup: a second T3 server on its own port and base dir, so the normal server was untouched. It ran one OpenCode thread whose first turn runs sleep 600, which keeps the turn active while the server is otherwise idle.

    Steps:

    1. Start the isolated server:
      t3 serve --mode web --port <port> --host 127.0.0.1 --no-browser --base-dir <base-dir>
      
    2. A fresh base dir has the OpenCode provider disabled. Enable it in <base-dir>/userdata/settings.json before starting the thread:
      {"providerInstances":{"opencode":{"driver":"opencode","enabled":true,"config":{"binaryPath":"opencode"}}}}
    3. Add a project and start the thread:
      t3 project add <workspace> --base-dir <base-dir>
      t3 thread start <workspace> "Run this exact command and wait for it to finish: sleep 600" --base-dir <base-dir> --provider opencode --model opencode-go/deepseek-v4.1-flash
      
    4. Once the sleep 600 tool call is running, freeze the opencode serve that the isolated server started for 120 s, then resume it. <opencode-port> is the port T3 picked for it (visible in ps).
      pid=$(pgrep -f 'opencode serve --hostname=127.0.0.1 --port=<opencode-port>')
      kill -STOP "$pid"
      for i in $(seq 12); do sleep 10; awk '{print $3}' "/proc/$pid/stat"; done   # T every time
      kill -CONT "$pid"
      

    Result: the run failed at 02:23:31.761Z with transport_error, The OpenCode event stream was lost and could not reconnect., the same failure as the report. Times are UTC; reconnect start times and durations are from the server trace spans.

    02:21:55        SIGSTOP; /proc/<pid>/stat state T at every 10 s check
    02:22:38.239    reconnect 1   7758 ms  GET /api/event, no response headers
    02:22:48.006    reconnect 2  10010 ms
    02:23:00.017    reconnect 3  10005 ms
    02:23:09.633    reconnect 4  10018 ms
    02:23:21.651    reconnect 5  10026 ms
    02:23:31.679    "Could not reconnect to the OpenCode event stream; failing the session."
                    HttpClientError: Transport: timed out after 10s (GET .../api/event)
    02:23:31.703    provider session status=error
    02:23:31.761    run failed (transport_error)
    02:23:45        SIGCONT
    02:23:46.028    OpenCode sends six GET /api/event 200s within 8 ms: the original
                    stream (span 140548 ms) and the five abandoned reconnects (15-19 ms)
    02:24:04.450    T3 stops the server (SIGTERM)
    

    The process was alive the whole time (state T, never exited), and it answered every reconnect within a second of SIGCONT. reconcile never ran, so the #25596 recovery never applied.

    The memory-throttled cgroup from the original report is only how the server got frozen in the real incident. Any freeze of the owned process longer than about 100 s reproduces this: at most 45 s of silence, then five 10 s subscribes with 2 s gaps (45 + 5×10 + 4×2 = 103 s).

    Why I think this is a real bug and not a no-repro:

    Could this be reopened?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions