[Bug]: A child that ignores SIGTERM never lets ProcessRunner's timeout return, leaking a VcsProcess permit; after 8 leaks every git call and every new turn hangs forever #20419

Before submitting

• I searched existing issues and did not find a duplicate.
• I included enough detail to reproduce or investigate the problem.

Related: #17209 (same trigger: glab auth status hangs while a configured host is offline), #16503 (unknown-remote probes saturate VcsProcess), #16636 (one stuck provider stalls the effect workers), #12507 (child processes outlive the server). Those describe slowness that clears up eventually. This one is a permanent deadlock: the permits never come back, and only a server restart recovers.

Area

apps/server

Steps to reproduce

1. Configure glab with a host that is only reachable over a VPN (here gitlab.corp.example, a placeholder for the real host), plus gitlab.com.
2. Turn the VPN off. Running glab auth status in a terminal now blocks indefinitely (still running after 5 minutes) and ignores SIGTERM (still alive 120s after kill -TERM; only SIGKILL stops it). This is glab 1.107.0 on Linux.
3. Run t3 serve and use the app normally. Source-control discovery runs glab auth status with the 5s probe timeout every time settings or the source-control/PR views refresh.
4. After the probe has timed out 8 times, which took about 45 minutes here, send a message in any thread.

A smaller repro without glab should be any ProcessRunner.run call (via VcsProcess.run) on a child that traps SIGTERM, e.g. sh -c 'trap "" TERM; sleep 600' with timeoutMs: 1000. I expect it never to return, based on the code below. I did not run this variant.

Expected behavior

A process that ignores SIGTERM is escalated to SIGKILL after a grace period. The timeout returns VcsProcessTimeoutError in about timeoutMs plus the grace period, and the VcsProcess permit is released.

Actual behavior

The message is accepted, the run moves to starting, and nothing else happens. ClaudeAdapterV2.startTurn never runs, and the provider CLI never receives the prompt (its stdin socket stays empty). Interrupting the run works, but every new message hangs the same way. Killing the provider process doesn't help. Only restarting t3 serve recovers, and the server then needs SIGKILL to exit.

Root cause (from the trace and the code at v0.1.1-nightly.20261016.3218):

1. processRunner.ts finalizeRunProcess implements timeouts as Effect.scoped + Effect.timeoutOption. On timeout the inner fiber is interrupted, and the timeout only returns after that fiber's finalizers finish. One of them is the spawner's scope release.
2. processRunner.ts runProcessCore spawns with no forceKillAfter. In @effect/platform-node-shared 4.0.5, NodeChildProcessSpawner terminateProcessGroup then does:

   yield* signalGroup(options?.killSignal ?? "SIGTERM")
   if (Predicate.isUndefined(options?.forceKillAfter)) {
     yield* awaitProcessExit(childProcess, exitSignal, processGroupGraceMillis) // bounded, 1s
   } else { /* SIGKILL escalation */ }
   yield* Deferred.await(exitSignal) // unbounded

   The module docs say that without forceKillAfter "the wait is limited to one second and never escalates". The final Deferred.await(exitSignal) has no bound, so a child that ignores SIGTERM keeps the release open forever. The bundled server contains this exact code.
3. VcsProcess.run holds one of 8 global permits (VCS_PROCESS_CONCURRENCY = 8) around runUnbounded. That permit is never released.
4. After 8 leaks, every VcsProcess.run waits forever on the semaphore. The turn-start path runs VcsDriverRegistry.detect → detectRepository (checkpoint baseline) before ClaudeAdapterV2.startTurn, so no turn on any thread can start.
5. The leaked glab processes run in their own process group and are reparented to init when the server dies, so they survive restarts. I found 17 of them on the machine, the oldest 6h17m old. They matched two full sets of 8 leaks from two server lifetimes, plus 1 leak already in the new server.

Other call sites already pass forceKillAfter (headlessChrome.ts, CodexProvider.ts, AntigravityInstallation.ts, NativeTelemetryClient.ts). ProcessRunner doesn't, and every VcsProcess and source-control CLI call goes through it.

Suggested fix

1. In runProcessCore, pass forceKillAfter (for example 2 seconds) to ChildProcess.make, so a timed-out child is SIGKILLed and the scope release finishes.
2. Consider releasing the VcsProcess permit when the timeout fires rather than after cleanup. Also consider giving discovery and auth probes their own budget, so slow hosting CLIs can't starve the git calls that the turn start and connection setup depend on.
3. Bound the VCS detect on the turn-start path. If the checkpoint baseline can't be read, start the turn without it rather than leaving the run in starting forever.
4. Possibly upstream in Effect: either bound the final Deferred.await(exitSignal) in terminateProcessGroup or correct the doc comment.
5. [Bug]: GitLab unknown-remote probe needs the network: glab auth status hangs when a configured host is offline (VPN), stalling vcs.refreshStatus for 65s #17209's fixes (an offline remoteRefinementArgs, --hostname, negative caching) remove this trigger, but any CLI that ignores SIGTERM can still cause the leak.

Impact

Major degradation or frequent failure. While the VPN is off, the server deadlocks permanently after a few dozen minutes of normal use. Every thread stops accepting turns, with no error shown. It recurs after each restart.

Version or commit

0.1.1-nightly.20261016.3218 (t3 serve, launched by the macOS desktop app over SSH, "Desktop-managed SSH")

Environment

• Server: Ubuntu 26.04.1 LTS, Linux 7.0 x64. Provider: Claude Code CLI (claudeAgent).
• glab 1.107.0 with hosts gitlab.com and gitlab.corp.example. The second is reachable only over VPN, and the VPN was off. jj is not installed.

Logs or stack traces

# All 8 VcsProcess permits leaked; each leak is a glab runProcessCore whose parent span never ended
# (VcsProcess.runUnbounded never closed). Times local, server started 19:02:48.
processRunner.runProcessCore glab  19:06:28.605  5001 ms  Interrupted  (parent span never ended)
processRunner.runProcessCore glab  19:09:05.631  2970 ms  Interrupted  (parent span never ended)
processRunner.runProcessCore glab  19:09:09.439  5001 ms  Interrupted  (parent span never ended)
processRunner.runProcessCore glab  19:15:33.228  4999 ms  Interrupted  (parent span never ended)
processRunner.runProcessCore glab  19:17:36.681  5001 ms  Interrupted  (parent span never ended)
processRunner.runProcessCore glab  19:24:19.937  5001 ms  Interrupted  (parent span never ended)
processRunner.runProcessCore glab  19:26:05.388  5000 ms  Interrupted  (parent span never ended)
processRunner.runProcessCore glab  19:47:36.931  5000 ms  Interrupted  (parent span never ended)
# Last VcsProcess.runUnbounded that ever started: 19:47:36.763. None after it, until the restart at 20:13.

# Turn that hung (run 8). Compare with run 7, which worked 18 min earlier.
19:50:12.775 orchestrationV2.providerTurnStart.start       Success 20 ms
19:50:12.789   orchestrationV2.EventSink.writeIfRunCurrent  Success 4 ms
19:50:12.795 VcsDriverRegistry.detect                      Interrupted 1060950 ms
19:50:12.795   detectRepository                            Interrupted 1060950 ms
19:50:12.795     VcsProcess.run  (no runUnbounded child)   Interrupted 1060950 ms
# run 7 at 19:31:48: detect → 3 git calls ~12 ms each → hasCheckpointRef → ClaudeAdapterV2.startTurn

# statev2.sqlite orchestration_v2_effect_outbox for the thread
provider-turn.start run 8   attempt_count=1  cancelled 17m41s later: "Run ... was interrupted before its provider turn started."
provider-turn.start run 9   attempt_count=1  cancelled: same
provider-turn.start run 10  attempt_count=1  cancelled: "Cancelled because the server restarted before the provider work completed."
# No provider turn row exists for runs 8–10. Run 11, after the restart, worked normally.

# Same thing in the previous server lifetime
VcsProcess.run (no runUnbounded child) 18:30:16 → 19:01:16  1860 s  Interrupted   (detectRepository)

# Leftover processes on the host before cleanup (ppid 1, own pgid)
glab auth status   x16, started 14:09–19:47 and reparented to init. Plus 1 child of the new server, started 20:14:58.

$ glab auth status >/dev/null 2>&1 & sleep 5; kill -TERM $!; sleep 120; kill -0 $! && echo alive
alive

Workaround: pkill -KILL -f '^glab auth status'. This frees the permits of the current server without a restart. Logging glab out of the VPN-only host also works, or put a glab wrapper on the server PATH that runs the real binary under timeout -s KILL.

The analysis came from server.trace.ndjson and statev2.sqlite. I can share sanitized excerpts if useful.