Skip to content

fix(dictation): cap each debug journal; bound the journal registry (#1276) - #1301

Merged
Juliusolsson05 merged 9 commits into
mainfrom
fix/dictation-journal-cap
Sep 25, 2026
Merged

Juliusolsson05 merged 9 commits into
mainfrom
fix/dictation-journal-cap

Conversation

@Juliusolsson05

@Juliusolsson05 Juliusolsson05 commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Fixes #1276. Related: #1299 (the recorder that ran 33 h, filed separately as P1).

Plan: docs/plans/2026-09-25-dictation-journal-cap.md (first commit; updated to match the code).

Evidence

One press produced a 1.196 GB dictation-debug file. By event, 96 % of it is per-chunk and per-sample: recorder:dataavailable, the CHUNK layer, deepgram:chunk:*, deepgram:message and AUDIO_LEVEL sample. Only 17 lifecycle events appear in the whole file. The only cleanup was a 14-day prune, and DictationDebugJournalRegistry.dispose had no callers.

Merged behaviour

  • Per-file budget of 16 MiB. Past it, high-frequency events (chunks, recorder data, Deepgram chunk and message events, audio samples, interim previews) are not written one by one. Instead they are counted into one journal:high-frequency-summary row per minute. Each row has per-kind counts, per-source last chunk indexes, maxPeak and nonZeroPeaks. A kept event (lifecycle, error, outcome, stop, final transcript) first closes the open summary window, so ordering around it is preserved. This keeps a Dictation can keep the microphone open for hours: no max duration, no silence stop (33 h recorded) #1299-style tail visible at minute resolution.
  • The registry keeps at most 64 journals and evicts the oldest. An evicted writer is flushed. flushAll() (the shutdown drain) waits for eviction flushes and for a drain already in flight.
  • Re-creating a writer for an evicted press keeps the budget. It starts from the larger of the on-disk size and every in-flight evicted writer's accounted bytes. It also keeps the press clock: tMs is anchored to the file's first {"ts","tMs"} line.
  • Known caveat: if the press is reopened before its first evicted writer has written anything, there is no file line to anchor on, and tMs restarts from the new writer's clock. ts and the byte budget are unaffected.

Tests

src/main/dictationJournal.test.ts covers:

  • every suppressed classification, and ERROR/OUTCOME preservation;
  • summary counts, peaks and minute cadence on disk, and lifecycle ordering;
  • oldest-versus-newest eviction, and eviction and shutdown with delayed writes;
  • in-flight evicted budgets and the clock anchor.

The new tests were red on main. npx tsc -b is clean. Review dispositions are in the comments.

🤖 Generated with Claude Code

Juliusolsson05 and others added 3 commits September 25, 2026 07:02
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…istry (#1276)

One press whose recorder ran 32.8 h (#1299) wrote a 1.196 GB journal, 96 %
of it per-chunk and per-sample events. Past 16 MiB those high-frequency
events are dropped (one marker says so); lifecycle, error and stop events
still land. The registry now keeps at most 64 journals, flushing and
evicting the oldest, since nothing ever disposed of them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Juliusolsson05 and others added 6 commits September 25, 2026 07:25
…budget, a stable budget, awaited eviction

- C / steering q29: one 'suppressed' marker erased the #1299 evidence (26 h
  of chunks and non-zero audio after deepgram:close). Past the budget the
  high-frequency events now fold into one summary line per minute (counts
  per kind, time span, last chunk index, peak stats), plus one before any
  kept event and on flush: ~430 KiB/day, bounded.
- A/C: a writer re-created for an evicted press restarted its budget; it
  now starts from the file's size.
- A/B: an eviction's flush was fire-and-forget and outside flushAll(); it
  is now awaited by the shutdown drain.
- A: TRANSCRIPT preview:interim counts as high-frequency.
- B: tests cover every classifier clause, ERROR/OUTCOME kept, exactly one
  marker, the #1299 tail, and both eviction paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…es, budget survives eviction, per-source chunk indexes

- A2/B1: drain returned at once while a timed write was in flight, so a
  flush (eviction, quit) could finish before the batch landed. Drains are
  now chained and flush awaits the chain; the draining flag is gone.
- A1: a writer re-created while the evicted one's write was in flight
  started from the smaller on-disk size. The registry keeps the evicted
  writer's accounted bytes until its flush lands.
- A3: renderer/main count chunks from 0 and Deepgram from 1, so one merged
  max invented a missing chunk; lastChunkIndex is now per layer/event.
- B2-B5: tests pin minute-scale summaries on disk before any flush, repeat
  counts and multiple peaks, the summary/close/summary order, the newest
  press never being evicted, and a blocked write at shutdown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ess counts toward the budget

One recorded count per press was wrong both ways: A's finalizer could
delete B's entry, and if B's small write landed first, B's own finalizer
removed the only entry while A's 16 MiB batch was still pending, so a
third writer started below the budget and logged raw chunks. The registry
now keeps each in-flight evicted writer's accounted bytes per press, takes
the max on reopen, and each finalizer removes only its own. Tested
A→B→C with both landing orders.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e press clock; plan matches the code

A writer re-created for an evicted press restarted tMs at 0, so hours-late
summaries sorted to the start of the press. It now anchors on the file's
first event (read from its ts/tMs prefix). The plan now describes the
per-minute summaries, the eviction budget and the chained drains instead
of the first draft's single marker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Juliusolsson05
Juliusolsson05 merged commit 54287d9 into main Sep 25, 2026
2 checks passed
@Juliusolsson05
Juliusolsson05 deleted the fix/dictation-journal-cap branch September 25, 2026 15:47
@Juliusolsson05

Copy link
Copy Markdown
Owner Author

Final disposition (posted after merge; steering q35 correction). I merged this PR at 54287d90 without first posting this comment, which is a process miss. From now on, the disposition is a pre-merge gate.

Tested head: 3d1fb302, which contains main 3c12bb73. quality-gate passed in run 36152352034.

Reviewer Round 1 Round 2 Final
A (correctness) FIX-BEFORE-MERGE FIX-BEFORE-MERGE fixed in 503ef71a, 2cd778e2
B (tests) FIX-BEFORE-MERGE FIX-BEFORE-MERGE fixed in 23557fa1, 503ef71a
C (debuggability/privacy) FIX-BEFORE-MERGE MERGE-READY (one minor) minor fixed in 2cd778e2

Material fixes from review:

  • Per-minute summaries replace the one-shot marker (b555c9eb, C1/A3). Past the 16 MiB budget, high-frequency events (chunks, audio samples, interim previews, Deepgram messages) are counted into journal:high-frequency-summary rows. Each row carries per-kind counts, maxPeak and nonZeroPeaks. A kept lifecycle event closes the open window first, so the Dictation can keep the microphone open for hours: no max duration, no silence stop (33 h recorded) #1299 tail stays visible.
  • The budget survives eviction (A1, b555c9eb / 23557fa1). A re-created writer starts from the on-disk size.
  • Each in-flight evicted writer counts (A-r2, steering q30, 503ef71a). A reopened press starts at the MAX of the in-flight writers' accounted bytes, so a write not yet on disk cannot reset the budget.
  • Shutdown joins eviction and in-flight drains (A2/B1, both rounds, b555c9eb / 23557fa1). flush() awaits a running drain and then drains the rest; flushAll() waits for writers still being disposed.
  • Per-source chunk indexes (A-r2 minor, 23557fa1). The summary no longer mixes the zero-based renderer/main indexes with the one-based Deepgram ones.
  • The press clock is kept (C-r2 minor, 2cd778e2). A re-created writer reads the original {"ts","tMs"} anchor from the file's first line, so summary tMs stays on the press clock.
  • Tests now cover:
    • every high-frequency classification, and ERROR/OUTCOME preservation;
    • exactly one summary per window, minute-scale timing on disk before the final flush, and repeated-event and multi-peak aggregation;
    • lifecycle-boundary ordering;
    • oldest-versus-newest eviction and the in-flight eviction and shutdown races.

Known caveat (confirmed by reading 54287d90): a press reopened before its first evicted writer has written anything to disk cannot find the original clock anchor. existingStartedAt reads only the file, and the in-flight ledger carries bytes, not the start time. That writer's tMs then restarts from its own clock. The byte budget still holds, because the in-flight bytes are counted. Timestamps (ts) stay absolute. This is not worth a follow-up unless a real journal shows it.

Not addressed: B-r1 minor 5 (no sanitized recorded *.dictation.jsonl fixture). The event shapes follow the current emitters.

🤖 Generated with Claude Code

Juliusolsson05 added a commit that referenced this pull request Sep 25, 2026
…osition gate, #369 needs-evidence

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dictation-debug journal has no per-file cap (1.2 GB file) and registry entries are never disposed

1 participant