bug(performance): heap watchdog retries a synchronous snapshot write every 2 s after a write failure #733

Problem

src/main/performance/heapWatchdog.ts writes a heap snapshot the first time main's used_heap_size crosses the trip line (min(1.5 GiB, 70% of the V8 limit)). writeHeapSnapshot is synchronous — a full GC plus graph serialisation, 3–15 s of hard freeze for the 190 MB dumps this machine produces — which is accepted as a one-shot forensic cost.

The catch block (~L211-218) resets snapshotWritten = false on a write failure so "a later sample can retry once the immediate pressure subsides". But once the heap is above 25% of the limit the sampler runs in fast mode every 2 s, and the failure that is most likely at that point — ENOSPC from a disk already holding 12 GB of debug artifacts, or an EIO — does not subside. The result is a synchronous multi-second writeHeapSnapshot every 2 s until the process dies or the disk frees up: a freeze loop under exactly the condition the subsystem creates.

Expected

Retry at most a small number of times with a real backoff (minutes, not the 2 s sample cadence), then give up for the rest of the run and record that it gave up.

Impact

Under a full disk the main process becomes unusable rather than merely slow; the heap-pressure signal that should help diagnose an OOM instead guarantees a freeze.

Refs #365, #364, #48, #103.