[Bug]: 0.1.10 startup migrations 82/83 take 17 min on a 32 GB statev2.sqlite while the server listens but answers nothing #35421

Before submitting

• I searched existing issues and did not find a duplicate.
• I included enough detail to reproduce or investigate the problem.

Related, not duplicates: #23372 (startup readiness gate makes requests hang with no timeout), #26591 (statev2.sqlite growth from inline payloads), #30065 (SQLite work on the server's event loop).

Area

apps/server

Steps to reproduce

1. Run a long-lived standalone server (t3 __service-launcher under systemd) whose statev2.sqlite has grown to 32.1 GB (page size 4096, freelist_count 0, so all of it is live data; most of it is orchestration_v2_projection_turn_items.payload_json).
2. Update from 0.1.9-nightly.20261110.4478 to 0.1.10-nightly.20261112.4580 and let the service restart.
3. Open the web app or reach the server from a linked environment.

Expected behavior

The server becomes usable within seconds, or it answers requests with a clear "migrating, N of M" status while it works. Long index builds over large tables run after startup (or incrementally), not inside the single startup transaction that blocks readiness.

Actual behavior

The server listened on its port but answered nothing for about 19 minutes (HTTP accept queue backed up to 145; HTTP and WebSocket RPC requests in the trace waited 439 s). Linked environments reported it as not answering through the relay. The process sat in uninterruptible I/O wait (D state, folio_wait_bit_common) and read about 20.7 GB from disk before it served.

Trace spans for this start:

No trace lines were written between the server starting and the migrations finishing, so from outside it looked hung. A t3 CLI command run during that window refused with ServerOwnsDatabaseError, which was the only sign it was busy.

Impact

Every thread on that server, and every agent driving it from other machines, was unreachable for the whole migration. Restarting would only have restarted the migration, so the only option was to wait without any progress signal. The storage under this server answers small random reads in about 0.8 ms, at roughly one read in flight, which is what made the index builds this slow. That is ordinary for a VM disk on network or ZFS-backed SSD storage.

Version or commit

0.1.9-nightly.20261110.4478 → 0.1.10-nightly.20261112.4580 (standalone CLI service on Linux x64). Three other installs updated in the same run (4–6.5 GB databases) were reachable again shortly afterwards; I did not time them.

Environment

Ubuntu VM (KVM, virtio disk, ext4 on LVM), 38 GB RAM with about 14 GB available for page cache, so the 32 GB database could not be cached.

Logs or stack traces

From server.trace.ndjson (spans above). pragma page_count × page_size = 32.1 GB, freelist_count = 0. iostat during the migration: about 2,100 reads/s, average read 18 KB, r_await 0.76 ms.

Suggestions, in case they help:

• Build new indexes after the server is ready (or in batches), so a large history doesn't block startup.
• Report migration progress in the trace or service log, and have requests that arrive during migration fail fast with a clear status instead of hanging.
• A supported way to keep statev2.sqlite from growing without bound while keeping thread history (compressing old turn-item payloads, or archiving old turns to a second file).