Remote server update fails when the staged preflight exceeds 30 s on a loaded host, and the app shows only "SocketReadError" #28198

What happened

"Update everything" from the desktop app updated the app, one server and four providers. It reported SocketReadError: An error occurred during Read for two remote boot-service servers. Both stayed on the old version. A retry later, with the host idle, succeeded and left no staging files behind.

Diagnosis

The trace on one of the failing servers shows cloud.server_self_update.update failing after 235 s with ServerSelfUpdateError: Could not prepare t3@0.1.6-nightly.20261031.3977. The cause was PinnedRuntimeInstallError (running the staged service preflight), from ProcessTimeoutError: .../runtime/versions/.staging-.../t3 timed out after 30000ms.

The host was heavily loaded during the update: load 17 on 4 cores, I/O pressure about 34% over 5 minutes, swapping, event loop stalls of up to 12 s. PREFLIGHT_TIMEOUT in apps/server/src/cloud/selfUpdate.ts is a fixed 30 s.

About 38 s before that failure, the client's update socket dropped (ws.rpc.server.updateServerWithProgress interrupted, probably a ping timeout during an event loop stall). packages/client-runtime/src/connection/outdatedHostUpdate.ts only treats a dropped socket as success when it has a complete event, or progress plus boot-service/respawn. Here it surfaced the raw SocketReadError, so the user never sees the server's actual failure.

Suggestions: make the preflight timeout tolerate slow hosts, and after a socket drop, poll the server for the outcome of the update instead of reporting the socket error.

Steps to reproduce

1. Link a remote boot-service server on a slow or loaded Linux host (4 cores, 5 GB RAM, heavy disk I/O).
2. Run "Update everything" from the desktop app to a newer nightly.
3. The client reports SocketReadError: An error occurred during Read for that server; the server log shows the staged preflight timing out after 30000 ms, and the server stays on the old version.

Version

Client and target: 0.1.6-nightly.20261031.3977. Failing servers were on 0.1.6-nightly.20261030.3940.

Environment

Desktop app on Linux x64 (CachyOS). Remote servers: boot-service (systemd user unit) on Linux x64, linked over Tailscale HTTPS.

Evidence

Failing server trace (UTC):
- 16:36:37 `ws.rpc.server.updateServerWithProgress` starts
- 16:39:54 that span ends Interrupted (client socket dropped)
- 16:40:14 preflight `processRunner.runProcessCore` (t3) interrupted after 31.2 s
- 16:40:32 `cloud.pinned_runtime.ensure_installed` fails: `PinnedRuntimeInstallError: Pinned runtime install failed while running the staged service preflight. [cause]: ProcessTimeoutError: Process '.../.staging-sXvlYz/t3' timed out after 30000ms`

Service log around the same time: `event loop stalled for 12346 ms`. Only this server's logs were checked; the other failing server showed the same symptom.

Related issues

#17899, #18944

Fix applied or workaround

Retry the update when the host is idle (worked here), or run t3 update on the host.

Filed by

Claude Code (Claude Opus 5.5) via T3 Code triage