[Bug]: One slow SQLite write freezes the whole server #35727

Closed as a duplicate of #30065. The details are summarized there.

What happens

The server saves to its SQLite database from its main thread, the same thread that answers every request. When one save is slow because the disk is slow, the whole server stops answering until it finishes, and clients disconnect.

Area

packages/contracts or packages/shared

How to reproduce

Slow the disk down on purpose with strace (Linux, no root needed), making one in every 2,000 database writes take 3 extra seconds:

strace -f -qq --seccomp-bpf -o /dev/null -e trace=pwrite64 \
  -P "$DB" -P "$DB-wal" \
  -e inject=pwrite64:delay_enter=3000000:when=2000+2000 \
  node bench.ts

bench.ts opens the database the same way the server does and saves 500 small changes, 50 per second, while checking how late a 10 ms timer fires.

What should happen

A slow disk makes saves slower, but the server keeps answering.

What actually happens

On main (c846833), every stuck write froze the whole server for the full 3 seconds.

The same happens on a real dev server: with every 150th write stuck for 3 seconds and three Replay conversations running, 49 of 163 simple requests took over a second, the slowest 6 seconds. The server logged four freezes of 2 to 5.6 seconds.

On a real installation (Linux, 24 cores, NVMe, btrfs, many agents running) the server froze 146 times on 2026-11-12. 29 of those freezes lasted over 15 seconds, long enough to disconnect clients. The longest was 243 seconds. Single saves took up to 9.8 seconds.

Impact

Major degradation or frequent failure

Version

0.1.10-nightly.20261112.4580, and reproduced on main at c846833. Linux, Node 24.15, SQLite 3.53, 3.6 GB database.

Workaround

On btrfs, moving the database into a folder with copy-on-write turned off (chattr +C) made the file far less fragmented, which should make slow saves rarer. It doesn't stop one slow save from freezing the server.