Repository navigation
perf(worktree-activity): stream JSONL transcripts instead of readFile - #87
Merged
Juliusolsson05 merged 2 commits intoMay 13, 2026
Merged
Conversation
The dominant source of transient memory spikes in the main process is
the `readFile(path, 'utf8') + split('\n') + map(JSON.parse)` pattern
used in several JSONL-reading sites. For a 50 MB transcript the
pattern transiently keeps a 50 MB Buffer, a 50 MB string, a 50 MB
array of substrings, AND the parsed JS objects all live at once.
This helper wraps the existing createReadStream + readline pattern
already used in ghostJournal.ts so callers can replace those whole-file
reads with a one-line-at-a-time async iterable. Peak transient memory
drops from O(file_size) to O(longest_line).
Used by the next commit in this branch. Future commits should be able
to reuse this in sessionIndex and historyLoader the same way.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
parseTranscriptForActivity used to do
`readFile(path) → text.split('\n') → map(JSON.parse)` per transcript.
For a 50 MB Claude transcript that transiently kept the 50 MB Buffer,
50 MB JS string, the split-array of substrings (~50 MB), and the
parsed JS objects all live at once. With WorktreeActivityIndex's
60 s background refresh calling this for each candidate sequentially,
the per-tick peak was ~100-200 MB — visible in the system-perf
popover as a recurring spike-and-drop pattern every minute,
suspected primary cause of recent main-process OOMs.
Switching to streamJsonl drops the transient peak to one line at a
time. Same output shape (IndexedTranscript), same malformed-line
behaviour (silently skip), same call site contract. Verified
test:worktree-activity passes unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 13, 2026
Closed
This was referenced Sep 3, 2026
Closed
Juliusolsson05
added a commit
that referenced
this pull request
Sep 4, 2026
The initial history load streamed the whole transcript through readline and JSON.parse to keep the last 120 records, and the older-history page forward-scanned from byte 0 to its anchor on every request. On a workspace restore that ran for every resumed session in the first seconds of boot: 25 loads read 252 MB in 10.7 s of main-thread time, one 89 MB rollout took 4.6 s to return 120 entries, and the parse contended with nine provider spawns. Both readers now walk the file newest-first in 256 KiB blocks and stop as soon as the window is complete (limit + 1 records, so hasMore keeps its exact meaning). Lines are split on newline bytes and decoded only once complete, so records larger than a block and multi-byte characters on a block boundary come back exactly as the forward parse produced them. totalEntries is the tail's parsed records plus a raw newline count of the untouched head, which is exact for provider-written files (their only malformed line is the mid-append one at EOF, inside the parsed tail). The older page anchors on the newest occurrence of a duplicated marker, the one at the edge of a window that grew from the tail. Parse-error tolerance, the per-load intern pool, the missing-marker tail fallback and the empty-chunk read-failure path are unchanged. Fixes #747 Refs #87, #103, #93 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kk16MNVJtWAnCeGxGRuHqa
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The 60 s background refresh in
WorktreeActivityIndexwas callingparseTranscriptForActivityper discovered transcript, and that function usedreadFile(path, 'utf-8').then(t => t.split('\n').map(JSON.parse)). For a 50 MB JSONL transcript that pattern transiently kept four representations live: the 50 MB Buffer, the 50 MB JS string, the 50 MB-ish array of substrings, and the parsed objects. Sequentially across all candidates per refresh, this matched the 100–200 MB spike-and-drop pattern visible in the system-perf popover every ~60 seconds and is the most likely primary cause of the recent main-process OOM crashes the user has been seeing during normal work.This PR:
streamJsonlhelper insrc/shared/runtime/that yields parsed JSONL objects one line at a time usingcreateReadStream+readline.createInterface(same patternghostJournal.tsalready uses).readFile + split + mapbody ofparseTranscriptForActivitywith afor awaitloop overstreamJsonl.Peak transient memory per parsed transcript drops from O(file_size) to O(longest_line) — typically tens of KB per line even for large tool_use entries.
Out of scope (separate PRs)
src/main/sessionIndex.ts:287(palette search) andsrc/main/sessions/historyLoader.ts:121(session resume + pagination). Both will reusestreamJsonlin follow-up PRs.worktree-activity-index.jsonfile inindexStore.tsis single-JSON, not JSONL — streaming it requires a format migration shim or a JSON-stream parser dependency. Worth doing later, deliberately deferred from this PR.Test plan
npm run test:worktree-activitypasses (unchanged from baseline)npm run test:review-fixespassesnpm run build:appsucceedsgit diff main | grep -ic cc-shellreturns 0AGENT_CODE_PERF=1 npm run dev, open badge popover, observelarge_object_spaceover ~5 minutes. The 60 s spike pattern that was hitting 100–200 MB should drop substantially (target: <30 MB per refresh).Stats
2 files changed (+78 / -11).
🤖 Generated with Claude Code