From 558a68ee6253299d2b4d44df814732dd8cc40406 Mon Sep 17 00:00:00 2001 From: Julius Olsson Date: Thu, 7 May 2026 15:49:33 +0200 Subject: [PATCH 1/6] docs(ghost): findings + implementation plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Long-form analysis of how ghost works in cc-shell today, all four prior fix attempts and why each regressed, and the case table that drove the layered predicate landing in subsequent commits. Two docs: - 2026-05-07-ghost-system-findings.md — the diagnosis. Inventory of every component file, the dual-owner problem with SemanticStreamingTurn, the stale Phase 3 comment drift, the four fix attempts, and the corrected case table including the tail-sidecar gap an early single-rule predicate missed. - 2026-05-07-ghost-rendering-predicate.md — the superpowers plan for the layered predicate (timestamp gate + sidecar shape filter). Bite-sized tasks; this commit lands the docs. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../2026-05-07-ghost-rendering-predicate.md | 1345 +++++++++++++++++ .../plans/2026-05-07-ghost-system-findings.md | 908 +++++++++++ 2 files changed, 2253 insertions(+) create mode 100644 docs/superpowers/plans/2026-05-07-ghost-rendering-predicate.md create mode 100644 docs/superpowers/plans/2026-05-07-ghost-system-findings.md diff --git a/docs/superpowers/plans/2026-05-07-ghost-rendering-predicate.md b/docs/superpowers/plans/2026-05-07-ghost-rendering-predicate.md new file mode 100644 index 000000000..ed1d40617 --- /dev/null +++ b/docs/superpowers/plans/2026-05-07-ghost-rendering-predicate.md @@ -0,0 +1,1345 @@ +# Ghost Rendering Predicate Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Restore ghost rendering in the main feed, but only when JSONL has demonstrably stalled past the proxy AND the ghost is not a sidecar-shaped leak. Stops orphan ghosts from piling up at the bottom of feeds (today's regression) without losing the legitimate "JSONL went silent mid-turn" recovery case. + +**Architecture:** Layered predicate in `selectMergedEntries`: an orphan ghost renders iff its `_atp.updatedAt` is strictly newer than the newest JSONL entry timestamp this session has observed (`lastJsonlEntryAt`, new field on `SessionRuntime`) AND it does not match the known sidecar shape (single short text block in an assistant turn — title-gen / branch-name / predict-next-prompt fingerprint). Reuses atp's `mergeWithUpstream` for tail placement, which is correct in this case because surviving ghosts are by construction newer than every committed entry. The orphan TTL bumps from 3 s to 30 s so normal slow tool_results don't prematurely flip a ghost to render-eligible. + +**Tech Stack:** TypeScript, React 18, Zustand, Electron renderer; cc-shell's `src/renderer/src/workspace/{ghosts,mergedEntries,workspaceState}.ts` and `hook/ipc/useIpcSubscriptions.ts`; `agent-transcript-parser/ghost` for primitives. No atp changes needed. + +**Background reading:** [2026-05-07-ghost-system-findings.md](2026-05-07-ghost-system-findings.md) is the long-form analysis of how ghost works today, the four prior fix attempts, and why a single-rule predicate was insufficient. This plan implements the corrected layered predicate that came out of that analysis plus a cross-reading review. + +--- + +## Why two rules, not one + +The single-rule timestamp predicate (`updatedAt > lastJsonlEntryAt`) handles "stale orphan from earlier in the session below newer commits" but fails for "sidecar leak after the last real commit." Concretely: + +``` +real assistant JSONL committed at t=100 +predict-next-prompt sidecar at t=105 +no further JSONL writes (user walked away) + +ghost.updatedAt=105 > lastJsonlEntryAt=100 → predicate would render the sidecar at the tail +``` + +Production bundles show this is the dominant failure mode (predict-next-prompt fires after every real turn, accumulates at the tail with no later real turn to supersede via the timestamp comparison). So we keep the renderer-side sidecar-shape filter from `2a83978` as a backstop. + +Trade-off: a real assistant turn that crashed before any JSONL write AND was a single short text block (e.g. a "Done." reply) would also be hidden. Acceptable — that's a rare invisible loss vs. the everyday clutter of orphan title-gen / predict-next-prompt fragments. + +## File structure + +| File | Role | Change | +|---|---|---| +| `src/renderer/src/workspace/workspaceState.ts` | Runtime shape + factory | Add `lastJsonlEntryAt: number \| null` field, init `null` in `emptyRuntime()` | +| `src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts` | Live JSONL ingest, orphan sweep, ghost reducer wiring | Stamp `lastJsonlEntryAt` from appended entry timestamps; bump `GHOST_ORPHAN_TTL_MS` from 3000 to 30000 | +| `src/renderer/src/workspace/hook/actions/initialHistory.ts` | Bootstrap initial-history pull on resume | Stamp `lastJsonlEntryAt` from loaded entries' timestamps | +| `src/renderer/src/workspace/mergedEntries.ts` | Render-time selector | Replace body with the layered predicate; keep `ghostHasSidecarShape` from `2a83978`; restore `mergeWithUpstream` import | +| `src/renderer/src/workspace/ghosts.ts` | Renderer-side ghost reducer | Comment cleanup (file-level block) — describe ghost's actual job today, not the unfinished Phase 3 framing | +| `src/renderer/src/workspace/tile-tree/TileLeaf.tsx` | Call-site for selector | Comment cleanup — match the new behavior | +| `scripts/test-ghost-fallback.ts` | Existing regression script (no new test files) | Replace assertions with the new predicate's case matrix | + +`history.ts` (older-history pagination) is intentionally untouched — prepended entries are by construction older than what's already in `runtime.entries` and must not move `lastJsonlEntryAt` forward. + +DO NOT touch: +- `agent-transcript-parser/` — primitives are correct. +- `src/main/ghostJournal.ts`, `src/main/ipc/ghost.ts`, `src/preload/api/ghost.ts` — persistence layer. +- `reconcileUpstream`, `orphanStale`, `gcSupersededGhosts`, `ghostsFromSemanticTurn`, `ghostsToPersist` in `ghosts.ts` — reducers. +- `claudeSession.ts` committed-tool_result bridge — different problem. +- `ClaudeProxyAdapter.isSidecarFlow` and friends — proxy-side filter, complementary. + +## Branch + worktree convention + +Per `feedback_worktree_default.md` (`~/.claude/projects/.../memory/`), branch work goes in `.worktrees/`. Current state: existing local-only branch `fix/hide-orphan-ghost-tail` was developed on the main checkout, not in a worktree, with provisional changes that this plan supersedes. **Task 0** below moves to a properly-named worktree before code changes start. + +Per `reference_gh_account.md`, the cc-shell remote is on `Juliusolsson05`; verify `gh auth status` shows that account before any `gh pr create` step in Task 9. + +Per `feedback_no_test_bloat.md`, this plan modifies only `scripts/test-ghost-fallback.ts` (already exists) and does NOT add any new test files or wire any new `test:*` scripts. + +Per `feedback_no_auto_merge.md`, Task 9 stops at PR open. Do not auto-merge. + +--- + +## Task 0: Branch hygiene + +**Files:** +- No file changes; git operations only. + +- [ ] **Step 1: Verify branch state** + +Run from repo root: +```bash +git status --short --branch +git branch --show-current +git log --oneline @{u}..HEAD 2>/dev/null || echo "no upstream" +``` + +Expected: on `fix/hide-orphan-ghost-tail`, with modified files in `scripts/test-ghost-fallback.ts`, `src/renderer/src/workspace/mergedEntries.ts`, and `src/renderer/src/workspace/tile-tree/TileLeaf.tsx`. The "no upstream" output confirms the branch is local-only. + +- [ ] **Step 2: Reset the branch back to main and remove old changes** + +The provisional v1 changes on this branch are superseded by this plan. Remove them so we start clean. + +```bash +git checkout main +git branch -D fix/hide-orphan-ghost-tail +git checkout -- scripts/test-ghost-fallback.ts src/renderer/src/workspace/mergedEntries.ts src/renderer/src/workspace/tile-tree/TileLeaf.tsx 2>/dev/null || true +git status --short +``` + +Expected: clean working tree, on `main`, only `vendor/in_progress/` and `docs/superpowers/plans/2026-05-07-ghost-system-findings.md` may remain as untracked (those stay). + +- [ ] **Step 3: Create worktree on a new branch** + +```bash +git worktree add .worktrees/render-stuck-ghosts -b fix/render-stuck-ghosts main +cd .worktrees/render-stuck-ghosts +pwd +``` + +Expected: print `/.worktrees/render-stuck-ghosts`. All subsequent steps run from that working directory. + +- [ ] **Step 4: Verify gh account** + +```bash +gh auth status 2>&1 | head -10 +``` + +Expected: shows `Juliusolsson05` as the active account. If not, switch with `gh auth switch` before any later `gh` step. Don't fix this in Task 9 — fix it now so the rest of the plan is uninterrupted. + +--- + +## Task 1: Add `lastJsonlEntryAt` field to `SessionRuntime` + +**Files:** +- Modify: `src/renderer/src/workspace/workspaceState.ts` + +- [ ] **Step 1: Add the field to the type** + +Open `src/renderer/src/workspace/workspaceState.ts`. Find the `SessionRuntime` type at line 248. Add the new field immediately after the `ghosts` field (currently around lines 379-393). The full field, with a thick WHY comment per project CLAUDE.md: + +```ts + /** Wall-clock ms (epoch) of the newest JSONL entry timestamp we + * have observed for this session. + * + * WHY this exists separately from `entries[entries.length-1]`: + * Render decisions need a single comparable scalar against + * ghost `_atp.updatedAt`. Walking `entries` to find the max + * timestamp on every render would be O(N) per call; this is + * O(1) read after a single O(burst-size) update at ingest + * time. Equally important: the comparison must use entry + * *timestamp* (when the event happened on the original + * timeline), not `Date.now()`. On resume after a crash, the + * most recent JSONL entry might be from yesterday but ghost + * `updatedAt` is also from yesterday — both sides need to be + * "wall-clock when this was observed by the producer" or the + * comparison flips to nonsense. + * + * WHY null instead of 0 for "never seen": + * A 0-valued sentinel would make `ghost.updatedAt > 0` always + * true, accidentally rendering ghosts on a brand-new session + * that has produced nothing yet. Null is checked explicitly + * in `selectMergedEntries`. + * + * Used by `selectMergedEntries` (./mergedEntries.ts) to decide + * whether an orphaned ghost represents JSONL stalling past the + * proxy (render — proxy event is the only record) vs. a sidecar + * leak Claude Code never logs to its rollout (hide — JSONL kept + * writing real turns past it). See + * docs/superpowers/plans/2026-05-07-ghost-system-findings.md + * for the full rationale. */ + lastJsonlEntryAt: number | null + ghosts: Map +``` + +(Place `lastJsonlEntryAt` BEFORE `ghosts` to keep the related-render-state fields adjacent — `ghosts` and `lastJsonlEntryAt` are both consumed only by `selectMergedEntries`.) + +- [ ] **Step 2: Initialize in `emptyRuntime()`** + +Find `emptyRuntime()` at line 425. Add the new field initialization immediately before `ghosts: new Map()`: + +```ts + lastJsonlEntryAt: null, + ghosts: new Map(), +``` + +- [ ] **Step 3: Verify TypeScript compiles** + +Run: +```bash +npx tsc --noEmit -p tsconfig.web.json 2>&1 | head -30 +``` + +Expected: clean (no new errors). If there are errors about missing `lastJsonlEntryAt` from elsewhere, those callers need updating — note them and add to the relevant tasks below. + +- [ ] **Step 4: Commit** + +```bash +git add src/renderer/src/workspace/workspaceState.ts +git commit -m "$(cat <<'EOF' +runtime: add lastJsonlEntryAt to SessionRuntime + +Foundation for the ghost-rendering predicate. The field is set in +later commits from the JSONL ingest paths and consumed by +selectMergedEntries to distinguish "JSONL stalled past the proxy" +from "sidecar leak JSONL was never going to write." See +docs/superpowers/plans/2026-05-07-ghost-system-findings.md for the +full design rationale. + +No behavior change: nothing reads the field yet. + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +--- + +## Task 2: Stamp `lastJsonlEntryAt` from live JSONL ingest + +**Files:** +- Modify: `src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts:1102-1190` + +- [ ] **Step 1: Locate the runtime patch in `onSessionJsonlEntries`** + +Open `src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts`. Find the bulk JSONL ingest handler around line 846. Around line 1102, locate the block that builds `baseEntries` and immediately precedes the runtime patch (`const nextRuntimeBase = withDerivedSessionStatus(...)` at ~1171). The `appended` array is fully populated by this point. + +- [ ] **Step 2: Insert the timestamp scan** + +Immediately after the `baseEntries` calculation (around line 1109, before the "Ghost reconciliation" comment block), insert: + +```ts + // Track the newest JSONL entry timestamp this session has + // ever observed. selectMergedEntries uses this to decide + // whether an orphaned ghost is past the JSONL tail (render — + // proxy stalled past disk) or covered by it (hide — + // JSONL kept writing past this ghost, so it's a sidecar + // leak Claude Code never logs to its rollout). Comparison + // uses entry.timestamp (ISO 8601 on both Claude and Codex + // entries) NOT Date.now(), because on resume after a crash + // we want apples-to-apples wall-clock semantics with ghost + // _atp.updatedAt — both sides represent "when the producer + // observed this," so a yesterday-vs-yesterday comparison + // is valid even if "now" is hours later. + let lastJsonlEntryAt = current.lastJsonlEntryAt + for (const entry of appended) { + const ts = (entry as { timestamp?: unknown }).timestamp + if (typeof ts !== 'string') continue + const ms = Date.parse(ts) + if (!Number.isFinite(ms)) continue + if (lastJsonlEntryAt === null || ms > lastJsonlEntryAt) { + lastJsonlEntryAt = ms + } + } +``` + +- [ ] **Step 3: Add `lastJsonlEntryAt` to the runtime patch** + +Find the `nextRuntimeBase` block at approximately line 1171. Add the new field next to `ghosts: nextGhosts`: + +```ts + ghosts: nextGhosts, + lastJsonlEntryAt, +``` + +- [ ] **Step 4: Update the no-change guard** + +The no-op short-circuit at ~line 1152-1161 must include the new field — otherwise a JSONL burst that only updates the timestamp (e.g. an entry with no new ghost reconciliation) would silently fail to land. Replace the existing guard: + +```ts + const ghostsChanged = nextGhosts !== current.ghosts + const noChange = + appended.length === 0 && + reconciledOptimisticText === null && + pendingCompaction === current.pendingCompaction && + queuedMessages === current.queuedMessages && + awaitingAssistant === current.awaitingAssistant && + workContext === current.workContext && + workActivity === current.workActivity && + !ghostsChanged +``` + +with: + +```ts + const ghostsChanged = nextGhosts !== current.ghosts + const lastJsonlChanged = lastJsonlEntryAt !== current.lastJsonlEntryAt + const noChange = + appended.length === 0 && + reconciledOptimisticText === null && + pendingCompaction === current.pendingCompaction && + queuedMessages === current.queuedMessages && + awaitingAssistant === current.awaitingAssistant && + workContext === current.workContext && + workActivity === current.workActivity && + !ghostsChanged && + !lastJsonlChanged +``` + +- [ ] **Step 5: Run build** + +```bash +npm run build 2>&1 | tail -20 +``` + +Expected: clean build. The renderer + main + preload all transpile. + +- [ ] **Step 6: Commit** + +```bash +git add src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts +git commit -m "$(cat <<'EOF' +ipc: stamp lastJsonlEntryAt from live JSONL ingest + +Walks each burst's appended entries, parses the entry.timestamp ISO +string, and tracks the max as wall-clock-ms epoch on the runtime. +Fed to selectMergedEntries in a later commit to gate orphan ghost +rendering against the JSONL tail. + +No visible behavior change yet: the field is written but no +consumer reads it. + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +--- + +## Task 3: Stamp `lastJsonlEntryAt` from initial-history bootstrap + +**Files:** +- Modify: `src/renderer/src/workspace/hook/actions/initialHistory.ts:170-200` + +- [ ] **Step 1: Locate the initial-entries patch site** + +Open `src/renderer/src/workspace/hook/actions/initialHistory.ts`. Find the `setRuntimes` block around line 114; the runtime patch sets `entries:` around line 194. The `initialEntries` array is fully populated by this point. + +- [ ] **Step 2: Insert the timestamp scan and field** + +Immediately before the runtime patch (just before the `setRuntimes(prev => { const current = prev[sessionId]; if (!current) return prev; ... })` call's return value), compute the new value. Look for the line that builds the spread: + +Find the existing patch (around line 192-198): + +```ts + return { + ...prev, + [sessionId]: { + ...current, + entries: initialEntries.length > 0 + ? [...initialEntries, ...current.entries] + : current.entries, + historyOldestMarker: initialOldestMarker ?? current.historyOldestMarker, + hasOlderHistory: chunk.hasMore, +``` + +Insert the timestamp computation BEFORE the `return` statement, in the same `setRuntimes` callback scope: + +```ts + // Bootstrap-load equivalent of the live-ingest stamping in + // useIpcSubscriptions.ts. selectMergedEntries gates orphan + // ghost rendering against this timestamp; on resume we need + // it primed from the loaded JSONL tail so a ghost from the + // previous session whose updatedAt is OLDER than the + // freshest JSONL entry stays correctly hidden, while a + // ghost newer than every loaded entry (the + // "JSONL-stopped-mid-turn before the previous run died" + // case) surfaces as expected. + let lastJsonlEntryAt = current.lastJsonlEntryAt + for (const entry of initialEntries) { + const ts = (entry as { timestamp?: unknown }).timestamp + if (typeof ts !== 'string') continue + const ms = Date.parse(ts) + if (!Number.isFinite(ms)) continue + if (lastJsonlEntryAt === null || ms > lastJsonlEntryAt) { + lastJsonlEntryAt = ms + } + } +``` + +Then add the field to the returned patch: + +```ts + return { + ...prev, + [sessionId]: { + ...current, + entries: initialEntries.length > 0 + ? [...initialEntries, ...current.entries] + : current.entries, + historyOldestMarker: initialOldestMarker ?? current.historyOldestMarker, + hasOlderHistory: chunk.hasMore, + lastJsonlEntryAt, +``` + +- [ ] **Step 3: Verify history.ts (older-history pagination) is NOT touched** + +```bash +grep -n 'lastJsonlEntryAt' src/renderer/src/workspace/hook/actions/history.ts || echo "absent — correct" +``` + +Expected: "absent — correct". Older-history loads PREPEND entries that are by construction older than `runtime.entries`; touching the timestamp from this path would regress it. + +- [ ] **Step 4: Build** + +```bash +npm run build 2>&1 | tail -10 +``` + +Expected: clean. + +- [ ] **Step 5: Commit** + +```bash +git add src/renderer/src/workspace/hook/actions/initialHistory.ts +git commit -m "$(cat <<'EOF' +bootstrap: stamp lastJsonlEntryAt from initial-history load + +Mirrors the live-ingest stamping in useIpcSubscriptions: scans the +loaded JSONL tail, takes the max parseable entry.timestamp. +Necessary so that on resume the ghost-render predicate has the +right comparison anchor against ghost _atp.updatedAt loaded from +disk. + +Older-history pagination (history.ts) deliberately does NOT touch +this field — its prepended entries are by construction older than +what's already in runtime.entries. + +No visible behavior change yet. + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +--- + +## Task 4: Update `scripts/test-ghost-fallback.ts` for the new predicate (failing first) + +**Files:** +- Modify: `scripts/test-ghost-fallback.ts` + +This task writes the assertions BEFORE the implementation lands in Task 5. The test will fail until Task 5 is committed. + +- [ ] **Step 1: Replace the test file body** + +Open `scripts/test-ghost-fallback.ts`. Replace the entire file body (everything below the imports) with the new case matrix. Full file content: + +```ts +import assert from 'node:assert/strict' + +import { + createGhost, + orphanGhost, + supersedeGhost, + type ClaudeContentBlock, + type ClaudeEntry, +} from 'agent-transcript-parser/ghost' + +import { orphanStale } from '../src/renderer/src/workspace/ghosts' +import { selectMergedEntries } from '../src/renderer/src/workspace/mergedEntries' +import { emptyRuntime } from '../src/renderer/src/workspace/workspaceState' + +// Wall-clock anchor for tests. Concrete values keep the predicate +// inputs explicit and avoid Date.now() drift between assertions. +const T_OLD_JSONL = 1_700_000_000_000 +const T_GHOST_NEWER = T_OLD_JSONL + 5_000 +const T_GHOST_OLDER = T_OLD_JSONL - 5_000 + +function baseEntry(uuid: string, timestamp = '2026-04-24T00:00:00.000Z'): ClaudeEntry { + return { + type: 'assistant', + uuid, + parentUuid: null, + sessionId: 'session-1', + timestamp, + message: { + id: `msg-${uuid}`, + role: 'assistant', + content: [{ type: 'text', text: `committed ${uuid}` }], + }, + } as ClaudeEntry +} + +function ghostWithContent( + turnId: string, + content: ClaudeContentBlock[], + now: number, +) { + return createGhost({ + sessionId: 'session-1', + turnId, + blockIndex: 0, + role: 'assistant', + content, + now, + }) +} + +function shortTextGhost(turnId: string, text: string, now: number) { + return ghostWithContent(turnId, [{ type: 'text', text }], now) +} + +function longTextGhost(turnId: string, now: number) { + // 250 chars — past the SIDECAR_GHOST_TEXT_MAX threshold (200), so + // the sidecar shape filter will let it through. + const text = 'A real assistant turn that streamed enough text that it could not be a one-line title-gen sidecar leak. The shape filter is a 200-char cap; this string deliberately blows past that to verify the structural rule applies.' + return ghostWithContent(turnId, [{ type: 'text', text }], now) +} + +function toolUseGhost(turnId: string, now: number) { + // Tool_use ghosts are explicitly NOT sidecar-shaped (the shape + // filter only matches single-text-block assistant ghosts), so a + // tool_use orphan past lastJsonlEntryAt should render even if + // short. + return ghostWithContent( + turnId, + [ + { + type: 'tool_use', + id: `tool-${turnId}`, + name: 'Read', + input: { file_path: '/tmp/example.ts' }, + }, + ], + now, + ) +} + +// --------------------------------------------------------------------------- +// 1. Empty ghost map → return entries by identity (reference-stable +// short-circuit; Feed memos rely on this). +// --------------------------------------------------------------------------- +{ + const runtime = emptyRuntime() + runtime.entries = [baseEntry('real-1')] + const merged = selectMergedEntries(runtime, null) + assert.equal(merged, runtime.entries) +} + +// --------------------------------------------------------------------------- +// 2. Ghost not yet orphaned → hidden (JSONL might still arrive +// within TTL; SemanticStreamingTurn covers the live current +// turn, this ghost shouldn't double-render). +// --------------------------------------------------------------------------- +{ + const runtime = emptyRuntime() + runtime.entries = [baseEntry('real-1')] + runtime.lastJsonlEntryAt = T_OLD_JSONL + const ghost = longTextGhost('turn-a', T_GHOST_NEWER) + runtime.ghosts = new Map([[ghost.uuid, ghost]]) + const merged = selectMergedEntries(runtime, null) + assert.equal(merged, runtime.entries, 'unorphaned ghost must not render') +} + +// --------------------------------------------------------------------------- +// 3. Superseded ghost → hidden (JSONL caught up; reconcileUpstream +// has the row already). +// --------------------------------------------------------------------------- +{ + const runtime = emptyRuntime() + runtime.entries = [baseEntry('real-1')] + runtime.lastJsonlEntryAt = T_OLD_JSONL + const ghost = orphanGhost( + supersedeGhost(longTextGhost('turn-a', T_GHOST_NEWER), 'real-1', T_GHOST_NEWER + 1), + T_GHOST_NEWER + 2, + ) + runtime.ghosts = new Map([[ghost.uuid, ghost]]) + const merged = selectMergedEntries(runtime, null) + assert.equal(merged, runtime.entries, 'superseded ghost must not render') +} + +// --------------------------------------------------------------------------- +// 4. Orphan ghost for the live current turn → hidden. +// SemanticStreamingTurn owns the live turn render; surfacing a +// ghost for the same turn would double-render. +// --------------------------------------------------------------------------- +{ + const runtime = emptyRuntime() + runtime.entries = [baseEntry('real-1')] + runtime.lastJsonlEntryAt = T_OLD_JSONL + const ghost = orphanGhost( + longTextGhost('turn-a', T_GHOST_NEWER), + T_GHOST_NEWER + 1, + ) + runtime.ghosts = new Map([[ghost.uuid, ghost]]) + const merged = selectMergedEntries(runtime, 'turn-a') + assert.equal(merged, runtime.entries, 'current-turn ghost must not render') +} + +// --------------------------------------------------------------------------- +// 5. Orphan ghost OLDER than lastJsonlEntryAt → hidden. This is the +// "stale orphan from earlier in the session, JSONL kept writing +// past it" case: a sidecar leak that happened before later real +// JSONL entries. Predicate-4 catches it structurally. +// --------------------------------------------------------------------------- +{ + const runtime = emptyRuntime() + runtime.entries = [baseEntry('real-1')] + runtime.lastJsonlEntryAt = T_OLD_JSONL + const ghost = orphanGhost( + longTextGhost('turn-a', T_GHOST_OLDER), + T_GHOST_OLDER + 1, + ) + runtime.ghosts = new Map([[ghost.uuid, ghost]]) + const merged = selectMergedEntries(runtime, null) + assert.equal(merged, runtime.entries, 'orphan older than JSONL tail must not render') +} + +// --------------------------------------------------------------------------- +// 6. Orphan ghost NEWER than lastJsonlEntryAt with sidecar shape +// (single short text block, ≤200 chars) → hidden. This is the +// tail-sidecar case the timestamp predicate alone cannot +// catch: predict-next-prompt fires after the last real JSONL +// entry, no later real turn supersedes it. Shape filter is the +// backstop. +// --------------------------------------------------------------------------- +{ + const runtime = emptyRuntime() + runtime.entries = [baseEntry('real-1')] + runtime.lastJsonlEntryAt = T_OLD_JSONL + const ghost = orphanGhost( + shortTextGhost('turn-a', 'short ack', T_GHOST_NEWER), + T_GHOST_NEWER + 1, + ) + runtime.ghosts = new Map([[ghost.uuid, ghost]]) + const merged = selectMergedEntries(runtime, null) + assert.equal(merged, runtime.entries, 'sidecar-shaped tail orphan must not render') +} + +// --------------------------------------------------------------------------- +// 7. Orphan ghost NEWER than lastJsonlEntryAt with substantive text +// (>200 chars) → renders. This is the main case ghost was built +// for: JSONL stopped writing while proxy kept going for an +// in-flight assistant turn. +// --------------------------------------------------------------------------- +{ + const runtime = emptyRuntime() + runtime.entries = [baseEntry('real-1')] + runtime.lastJsonlEntryAt = T_OLD_JSONL + const ghost = orphanGhost( + longTextGhost('turn-a', T_GHOST_NEWER), + T_GHOST_NEWER + 1, + ) + runtime.ghosts = new Map([[ghost.uuid, ghost]]) + const merged = selectMergedEntries(runtime, null) + assert.equal(merged.length, 2, 'substantive orphan past JSONL tail must render') + assert.equal(merged[1]?.uuid, ghost.uuid) +} + +// --------------------------------------------------------------------------- +// 8. Orphan tool_use ghost NEWER than lastJsonlEntryAt → renders +// even though short. tool_use is structurally not a sidecar +// shape — Claude Code's auxiliary calls all return text-only +// bodies, never tool_use blocks. +// --------------------------------------------------------------------------- +{ + const runtime = emptyRuntime() + runtime.entries = [baseEntry('real-1')] + runtime.lastJsonlEntryAt = T_OLD_JSONL + const ghost = orphanGhost( + toolUseGhost('turn-a', T_GHOST_NEWER), + T_GHOST_NEWER + 1, + ) + runtime.ghosts = new Map([[ghost.uuid, ghost]]) + const merged = selectMergedEntries(runtime, null) + assert.equal(merged.length, 2, 'tool_use orphan past JSONL tail must render') + assert.equal(merged[1]?.uuid, ghost.uuid) +} + +// --------------------------------------------------------------------------- +// 9. lastJsonlEntryAt === null (fresh session, no JSONL ever) AND +// orphan ghost present → render decision falls through the +// timestamp gate (rule 4 only fires when lastJsonlEntryAt is +// non-null). The shape filter still applies, so the ghost +// renders only if it's not sidecar-shaped. +// --------------------------------------------------------------------------- +{ + const runtime = emptyRuntime() + runtime.entries = [baseEntry('real-1')] + runtime.lastJsonlEntryAt = null + const ghost = orphanGhost( + longTextGhost('turn-a', T_GHOST_NEWER), + T_GHOST_NEWER + 1, + ) + runtime.ghosts = new Map([[ghost.uuid, ghost]]) + const merged = selectMergedEntries(runtime, null) + assert.equal(merged.length, 2, 'orphan with null lastJsonlEntryAt must render if shape passes') + assert.equal(merged[1]?.uuid, ghost.uuid) +} + +// --------------------------------------------------------------------------- +// 10. orphanStale TTL behavior. Confirms the helper still produces +// reference-stable no-ops and that an orphan flag is set when +// the threshold elapses. +// --------------------------------------------------------------------------- +{ + const fresh = shortTextGhost('turn-a', 'fresh', T_OLD_JSONL) + const freshMap = new Map([[fresh.uuid, fresh]]) + assert.equal(orphanStale(freshMap, fresh._atp.updatedAt + 500, 1000), freshMap) + + const staleMap = orphanStale(freshMap, fresh._atp.updatedAt + 2000, 1000) + assert.notEqual(staleMap, freshMap) + assert.ok(staleMap.get(fresh.uuid)?._atp.orphanedAt !== undefined) +} + +console.log('ghost render predicate tests passed') +``` + +- [ ] **Step 2: Run the test (it MUST fail at this point)** + +```bash +npm run test:ghost-fallback 2>&1 | tail -30 +``` + +Expected: assertion failure on case 7 or 8 (the rendering cases). The current `selectMergedEntries` returns `entries` unconditionally, so cases that expect `merged.length === 2` will fail. + +If the test passes here, something is wrong — Task 5 hasn't been done yet so the predicate cannot be returning the expected behavior. Stop and investigate. + +- [ ] **Step 3: Commit the failing test** + +```bash +git add scripts/test-ghost-fallback.ts +git commit -m "$(cat <<'EOF' +test: spec the ghost-render predicate matrix (currently failing) + +10 cases covering the layered predicate: empty/superseded/ +unorphaned/current-turn skips, the timestamp gate (orphan older +than JSONL tail = hide), the sidecar shape backstop (tail orphan +with single ≤200-char text block = hide), substantive text +rendering, tool_use rendering (not sidecar-shaped), fresh-session +null-lastJsonlEntryAt path, and orphanStale TTL semantics. + +The implementation lands in the next commit (selectMergedEntries +rewrite). Cases 7-9 fail at HEAD because the current selector +returns entries unconditionally. + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +--- + +## Task 5: Implement the layered predicate in `selectMergedEntries` + +**Files:** +- Modify: `src/renderer/src/workspace/mergedEntries.ts` + +- [ ] **Step 1: Replace the entire file** + +Open `src/renderer/src/workspace/mergedEntries.ts`. Replace its entire contents with: + +```ts +// Render-time selector: decide which (if any) ghost entries get +// merged into the rendered feed. +// +// ----------------------------------------------------------------------------- +// Where ghost rendering fits in cc-shell today +// ----------------------------------------------------------------------------- +// +// The live current turn is rendered by `SemanticStreamingTurn` +// directly off `runtime.semantic.currentTurn`, NOT through ghosts. +// JSONL writes catch up within ~100 ms via `reconcileUpstream` and +// supersede the ghost. Most ticks, the ghost map has no rendered +// output here at all. +// +// The one case ghost rendering exists for is JSONL stalling past +// the proxy. Two situations produce that: +// 1. Live-stuck: the agent process gets wedged or its writer +// backlogs while the proxy keeps emitting events. Eventually +// the orphan TTL fires and the ghost is the only record of +// what happened for that turn. +// 2. Resume-after-crash: ghost log on disk has events past the +// newest JSONL entry; JSONL never caught up before the +// previous run died. On bootstrap, that ghost surfaces as the +// lost partial turn. +// +// ----------------------------------------------------------------------------- +// The layered predicate +// ----------------------------------------------------------------------------- +// +// A ghost is render-eligible iff ALL of: +// 1. Not superseded. +// 2. Orphaned (TTL has elapsed without a JSONL match). +// 3. `turnId !== currentTurnId` (SemanticStreamingTurn owns that). +// 4. `_atp.updatedAt > lastJsonlEntryAt` (proxy state is past the +// JSONL tail; structurally distinguishes the live-stuck case +// from "ghost from earlier in the session that JSONL kept +// writing past"). +// 5. Not sidecar-shaped. +// +// Why rules 4 AND 5, not just rule 4: +// The timestamp predicate is structurally correct for "stale +// orphan from earlier in the session below newer commits," but +// it cannot tell apart these two TAIL cases: +// a) JSONL stopped mid-turn, ghost has the lost partial turn +// (should render). +// b) Last real JSONL entry committed at t=100, then a sidecar +// leak (predict-next-prompt / title-gen / branch-name) at +// t=105 with no later real turn to supersede it (should +// NOT render). +// Both have ghost.updatedAt > lastJsonlEntryAt and produce a +// tail orphan with no JSONL counterpart. Rule 5 is a structural +// shape check that matches Claude Code's known sidecar +// fingerprint: assistant role + single text block + ≤200 chars. +// Predict-next-prompt with full conversation history can exceed +// the proxy-side budget predicate, but the response body is +// short and shaped this way; matching at render time is the +// backstop. +// +// Trade-off (knowingly accepted): a real assistant turn that +// crashed before any JSONL write AND was a single short text +// block (e.g. "Done.") would also be hidden. In production, that +// loss is rare; the alternative (orphan title-gen / next-prompt +// fragments piling up at the bottom of every session) is a daily +// UX harm. The trade is the same one `2a83978` made; this commit +// keeps the shape filter while adding the timestamp gate that +// `686b94e` was missing. +// +// ----------------------------------------------------------------------------- +// Reference stability (load-bearing for Feed memos) +// ----------------------------------------------------------------------------- +// +// Feed's row memos key off the `entries` array IDENTITY. When no +// ghost survives the predicate, this selector returns +// `runtime.entries` by identity, NOT a fresh `[...entries]`. The +// pre-fix `mergeWithUpstream` always returned `[...upstream, +// ...trailing]` even when `trailing` was empty, busting memos on +// every tick. atp's `mergeWithUpstream` is only called when +// `visible.size > 0`. +// +// Future work, not done here: +// Phase 3 of the original headless redesign (delete +// SemanticStreamingTurn, render the live current turn through +// ghosts + ordered insertion in `mergeWithUpstream`) requires +// atp learning to anchor by parentUuid / turnId / nearest +// committed neighbor instead of always tail-appending. Until +// that ships, the live current turn stays owned by +// SemanticStreamingTurn and ghost rendering is reserved for the +// JSONL-stalled-past-proxy fallback case described above. + +import type { Entry } from '@shared/types/transcript' +import type { SessionRuntime } from '@renderer/workspace/workspaceState' +import type { ClaudeContentBlock, GhostEntry } from 'agent-transcript-parser/ghost' +import { mergeWithUpstream } from 'agent-transcript-parser/ghost' + +// Cap chosen empirically against debug bundle 2026-05-07T08-26-35 +// (max sidecar turn = 41 chars, min real assistant turn = 76 +// chars). 200 leaves headroom for slightly-longer next-prompt +// variants and a generous safety margin before we'd start cutting +// into real prose. The same constant lived in mergedEntries.ts at +// commit 2a83978; reused here with the same threshold so the +// regression surface is unchanged. +const SIDECAR_GHOST_TEXT_MAX = 200 + +function ghostHasSidecarShape(ghost: GhostEntry): boolean { + // Only assistant orphans match. Sidecars come back through the + // proxy as assistant streams; user entries don't ghost (they get + // optimistic-row reconciliation separately) and don't have this + // failure mode. + if (ghost.message?.role !== 'assistant') return false + const content = ghost.message?.content + if (!Array.isArray(content)) return false + // Single block. Real assistant turns from a healthy proxy stream + // nearly always carry at least a tool_use companion or + // fragmentation across blocks even when short; a lone text block + // is the title-gen / predict-next-prompt fingerprint. + if (content.length !== 1) return false + const block = content[0] as ClaudeContentBlock + if (block.type !== 'text') return false + // `text` is required on text blocks per atp's content type, but + // be defensive — a malformed ghost should not crash the feed + // selector. + const text = (block as { text?: unknown }).text + if (typeof text !== 'string') return false + return text.length <= SIDECAR_GHOST_TEXT_MAX +} + +/** + * Render-time merge of `runtime.entries` with the surviving ghost + * set. See the file-level WHY block above for the predicate + * design. + */ +export function selectMergedEntries( + runtime: SessionRuntime, + currentTurnId: string | null, +): Entry[] { + const { ghosts, entries, lastJsonlEntryAt } = runtime + if (ghosts.size === 0) return entries + + const visible = new Map() + for (const [uuid, ghost] of ghosts) { + if (ghost._atp.supersededBy !== undefined) continue + if (ghost._atp.orphanedAt === undefined) continue + if (currentTurnId !== null && ghost._atp.turnId === currentTurnId) continue + // Rule 4: only ghosts past the JSONL tail. Null + // lastJsonlEntryAt (fresh session, never observed any JSONL + // entry) falls through — rule 5 still applies, and the rest + // of the predicate (orphaned + non-current) keeps this narrow. + if (lastJsonlEntryAt !== null && ghost._atp.updatedAt <= lastJsonlEntryAt) continue + if (ghostHasSidecarShape(ghost)) continue + visible.set(uuid, ghost) + } + if (visible.size === 0) return entries + + // Tail-append is correct for the surviving set: every visible + // ghost is by predicate-3 not the active turn and by predicate-4 + // newer than every committed entry, so chronologically it + // belongs at the very end. + return mergeWithUpstream(entries, visible, { + trustSupersededFlag: true, + }) as Entry[] +} + +/** + * Whether Feed should render the live `SemanticStreamingTurn` + * component. True iff there is a current turn — full stop. The + * merged feed selector (`selectMergedEntries`) hides ghosts whose + * `turnId === currentTurnId`, so SemanticStreamingTurn has + * exclusive ownership of the live view and there is no + * duplicate-render risk. + */ +export function shouldShowSemanticStreaming(runtime: SessionRuntime): boolean { + return runtime.semantic.currentTurn !== null +} +``` + +- [ ] **Step 2: Run the test (must now pass)** + +```bash +npm run test:ghost-fallback 2>&1 | tail -10 +``` + +Expected output ends with: +``` +ghost render predicate tests passed +``` + +If any case fails, re-read it; the predicate might be wrong (fix the implementation) or the test might have a typo (fix the test). Don't edit the test to match a buggy implementation. + +- [ ] **Step 3: Build** + +```bash +npm run build 2>&1 | tail -10 +``` + +Expected: clean. + +- [ ] **Step 4: Commit** + +```bash +git add src/renderer/src/workspace/mergedEntries.ts +git commit -m "$(cat <<'EOF' +fix(feed): render orphan ghosts only when JSONL has stalled past proxy + +Implements the layered predicate from +docs/superpowers/plans/2026-05-07-ghost-system-findings.md: + + 1. not superseded + 2. orphaned (TTL elapsed) + 3. turnId != currentTurnId (SemanticStreamingTurn owns the live) + 4. updatedAt > lastJsonlEntryAt (proxy state past JSONL tail) + 5. not sidecar-shaped (≤200-char single text block in + assistant turn — title-gen / predict-next-prompt fingerprint) + +The timestamp gate (rule 4) handles the "stale orphan from earlier +in the session below newer commits" case structurally. The shape +filter (rule 5) is the backstop for the tail-sidecar case where a +sidecar leak fires after the last real JSONL entry and no later +real turn supersedes it via timestamp comparison. + +Trade-off (accepted): a real assistant turn that crashed before +any JSONL write AND was a single short text block ("Done.") would +also be hidden. Rare invisible loss vs. daily clutter from orphan +title-gen / predict-next-prompt fragments. + +Test cases land in scripts/test-ghost-fallback.ts (modified +in-place; no new test files per project policy). + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +--- + +## Task 6: Bump the orphan TTL + +**Files:** +- Modify: `src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts:97` + +- [ ] **Step 1: Find the constant** + +Around line 97 of `src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts`: + +```ts +const GHOST_ORPHAN_TTL_MS = 3000 +const GHOST_ORPHAN_SWEEP_MS = 1000 +``` + +- [ ] **Step 2: Bump TTL and update its comment** + +Replace those two lines with: + +```ts +// Threshold before an unsuperseded ghost is marked orphaned. +// +// 3000ms was the value used while orphan rendering was always-on +// (commit 686b94e) — the goal was to surface fallback fast. With +// the layered predicate in mergedEntries.ts, an orphan flag merely +// makes a ghost ELIGIBLE for rendering; rules 4 and 5 still gate +// final visibility. So the TTL is now "how long do we wait before +// concluding JSONL had its chance for this ghost," not "how fast +// do we paint." 30000ms matches atp's library default and safely +// covers slow tool_results (large Read, large Bash output, slow +// fetches) without prematurely qualifying a ghost. Sweep cadence +// stays at 1s — that's the polling rate, not the threshold. +const GHOST_ORPHAN_TTL_MS = 30000 +const GHOST_ORPHAN_SWEEP_MS = 1000 +``` + +- [ ] **Step 3: Build** + +```bash +npm run build 2>&1 | tail -10 +``` + +Expected: clean. + +- [ ] **Step 4: Commit** + +```bash +git add src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts +git commit -m "$(cat <<'EOF' +ghosts: bump orphan TTL to 30s + +3s was set when orphan rendering was always-on and the goal was +"render fallback fast." With the layered render predicate in +mergedEntries.ts, the orphan flag merely qualifies a ghost as +RENDER-ELIGIBLE — rules 4 (timestamp gate) and 5 (shape filter) +still gate final visibility. So the TTL is now "how long before +we conclude JSONL had its chance," not "how fast we paint." + +30s matches atp's library default and safely covers slow tool +results (large Read, large Bash, slow fetches) without +prematurely qualifying a ghost mid-turn. Sweep cadence unchanged +at 1s — that's the polling rate. + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +--- + +## Task 7: Comment cleanup — remove stale Phase 3 / "live overlay" framings + +**Files:** +- Modify: `src/renderer/src/workspace/ghosts.ts` (file-level block) +- Modify: `src/renderer/src/workspace/tile-tree/TileLeaf.tsx` (call-site) + +- [ ] **Step 1: Rewrite `ghosts.ts` file-level block (lines 1-58)** + +Open `src/renderer/src/workspace/ghosts.ts`. Replace the file-level block comment (the long block at the top of the file, lines 1-58) with: + +```ts +// Renderer-side ghost reducer + the bridge between the live +// semantic stream, the durable JSONL transcript, and disk +// persistence. +// +// ----------------------------------------------------------------------------- +// What ghost is in cc-shell today +// ----------------------------------------------------------------------------- +// +// Ghost is a parallel disk-backed ledger of semantic events. As +// the proxy stream emits events, this file mints provisional +// `ClaudeEntry` records via atp's `createGhost`. When the +// authoritative JSONL entry lands (Claude's batched 100 ms drain; +// Codex's mpsc flush), `reconcileUpstream` matches by message.id / +// codexTurnId / tool_use_id and supersedes the ghost. If JSONL +// never matches (Claude Code's auxiliary calls — title gen, +// predict-next-prompt, branch-name gen — are not written to the +// rollout), `orphanStale` flags the ghost after the TTL. +// +// The live current turn renders via `SemanticStreamingTurn` +// directly off `runtime.semantic.currentTurn` — NOT through +// ghosts. So most ticks, the ghost map has no rendered output: +// the work this file does is bookkeeping (mint, reconcile, +// orphan, gc, persist) for two consumers: +// +// 1. `selectMergedEntries` (./mergedEntries.ts) — surfaces +// orphan ghosts ONLY when JSONL has stalled past the proxy +// (live-stuck mid-turn or resume-after-crash with partial +// JSONL). The layered predicate there has the full design +// rationale. +// +// 2. `ghostJournal.ts` (main process) — append-only JSONL log +// under /ghost-logs/.ghost.jsonl. +// Survives reload / restart so the JSONL-stuck case can +// recover the lost partial turn on resume via the bootstrap +// merge in src/renderer/src/workspace/hook/actions/session.ts. +// +// ----------------------------------------------------------------------------- +// Provider-aware reconciliation +// ----------------------------------------------------------------------------- +// +// Claude: upstream assistant entries carry `message.id == turnId`, +// so one upstream entry supersedes every ghost block for that +// turn at once. +// +// Codex: rollout emits one entry per content block, with the +// rollout response_id stamped onto the mapped entry by +// `stampCodexTurnId` in ../codex/rollout.ts. Match is by +// (turnId, blockIndex). When that fails, both providers fall back +// to tool_use_id / call_id pairing for tool blocks. +// +// ----------------------------------------------------------------------------- +// Reference stability +// ----------------------------------------------------------------------------- +// +// Every reducer below MUST return `prev` unchanged on no-op so +// React memoization holds at the call site. This isn't an +// optimization — it's load-bearing. The pre-fix versions always +// allocated `new Map(prev)` at the top, which made +// `nextGhosts !== current.ghosts` always true downstream, forcing +// a setRuntimes cascade that busted every useMemo([entries]) in +// Feed via selectMergedEntries. +// +// See `agent-transcript-parser/docs/ghost.md` for the underlying +// primitive's semantics, and +// docs/superpowers/plans/2026-05-07-ghost-system-findings.md for +// the longer story of how cc-shell got here. +``` + +- [ ] **Step 2: Update the `TileLeaf.tsx` call-site comment** + +Open `src/renderer/src/workspace/tile-tree/TileLeaf.tsx`. Find the block around lines 333-345 (the comment immediately before `entries={selectMergedEntries(...)}`). Replace it with: + +```tsx + // Committed transcript + (rare) orphan-ghost fallback. + // The layered predicate in selectMergedEntries renders + // a ghost only when JSONL has stalled past the proxy + // AND the ghost is not sidecar-shaped (title-gen / + // predict-next-prompt fingerprint). The live current + // turn is owned by `SemanticStreamingTurn` below; the + // `currentTurnId` argument hides any ghost for that + // turn so the two surfaces never double-render. +``` + +Then find the block around lines 363-375 (the comment immediately before `semanticTurn={...}`). Replace it with: + +```tsx + // Live-turn ownership: SemanticStreamingTurn renders + // the current turn end-to-end off the semantic channel. + // Ghosts for the same turnId are filtered out of the + // merged feed by selectMergedEntries (currentTurnId + // argument), so there is no double-render risk. + // shouldShowSemanticStreaming collapses to "is there a + // current turn?". +``` + +- [ ] **Step 3: Re-run the test (sanity)** + +```bash +npm run test:ghost-fallback 2>&1 | tail -5 +``` + +Expected: still passes (comment changes can't break logic). + +- [ ] **Step 4: Build** + +```bash +npm run build 2>&1 | tail -10 +``` + +Expected: clean. + +- [ ] **Step 5: Commit** + +```bash +git add src/renderer/src/workspace/ghosts.ts src/renderer/src/workspace/tile-tree/TileLeaf.tsx +git commit -m "$(cat <<'EOF' +docs: align ghost-system comments with current behavior + +Strips the "transcript-first feed with a ghost overlay" / "Phase 3 +will delete SemanticStreamingTurn" framings that described an +unfinished plan. Replaces with the actual current behavior: +ghost is a bookkeeping ledger that surfaces in the rendered feed +only when the layered predicate in selectMergedEntries fires +(JSONL stalled past proxy + not sidecar-shaped). The live current +turn is owned by SemanticStreamingTurn directly off the semantic +channel. + +No code change. The findings doc at +docs/superpowers/plans/2026-05-07-ghost-system-findings.md has +the long-form rationale. + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +--- + +## Task 8: Final verification + +**Files:** +- No file changes; verification only. + +- [ ] **Step 1: Run the regression test** + +```bash +npm run test:ghost-fallback 2>&1 | tail -5 +``` + +Expected: +``` +ghost render predicate tests passed +``` + +- [ ] **Step 2: Run the full build** + +```bash +npm run build 2>&1 | tail -20 +``` + +Expected: clean. Renderer / main / preload all transpile. + +- [ ] **Step 3: Quick code-search sanity checks** + +Confirm no leftover stale references: + +```bash +grep -rn 'Phase 3 will delete SemanticStreamingTurn' src/ packages/ 2>/dev/null +``` +Expected: no matches. + +```bash +grep -rn 'thin wrapper around atp' src/renderer/src/workspace/mergedEntries.ts +``` +Expected: no matches (the new file-level comment is the layered predicate explanation, not the wrapper framing). + +```bash +grep -n 'GHOST_ORPHAN_TTL_MS' src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts +``` +Expected: shows `30000` only. + +```bash +grep -n 'lastJsonlEntryAt' src/renderer/src/workspace/workspaceState.ts src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts src/renderer/src/workspace/hook/actions/initialHistory.ts src/renderer/src/workspace/mergedEntries.ts +``` +Expected: 4+ matches across all four files. + +- [ ] **Step 4: Verify history.ts was NOT touched** + +```bash +git diff main -- src/renderer/src/workspace/hook/actions/history.ts | head -5 +``` +Expected: empty (no diff against main). + +- [ ] **Step 5: Smoke test in the dev app (manual)** + +Run the dev app: + +```bash +npm run dev +``` + +In a Claude session: +1. Send a normal message. Confirm `SemanticStreamingTurn` paints the live response. Confirm no ghost row appears at the bottom of the feed. +2. After the turn completes, confirm no orphan ghost shows. (Title-gen / predict-next-prompt sidecars happen ~5s after each turn — wait 35s and confirm none appear.) +3. Open the FeedDebugPanel (`/cmd toggle-feed-debug-panel`) and confirm `ghost_orphan_sweep` entries fire periodically without the orphan rows appearing in the rendered feed. + +Expected: feed stays clean. Live turn renders via SemanticStreamingTurn. No bottom-of-feed ghost clutter. + +If you see a real stuck-mid-turn case (rare in normal usage), that's where the predicate WILL render an orphan ghost at the tail. That's correct. + +- [ ] **Step 6: Stop the dev app** + +`Ctrl-C` the `npm run dev` process. + +--- + +## Task 9: Open the PR + +**Files:** +- No file changes; git/gh operations. + +- [ ] **Step 1: Verify branch state** + +```bash +git status --short --branch +git log --oneline main..HEAD +``` + +Expected: 7 commits on `fix/render-stuck-ghosts` ahead of `main`. Working tree clean. + +- [ ] **Step 2: Push the branch** + +```bash +git push -u origin fix/render-stuck-ghosts +``` + +Expected: push succeeds, sets upstream to `origin/fix/render-stuck-ghosts` on `Juliusolsson05`. If gh / origin is on the wrong account, fix per Task 0 step 4 before retrying. + +- [ ] **Step 3: Open the PR** + +```bash +gh pr create --title "fix(feed): render orphan ghosts only when JSONL stalls past proxy" --body "$(cat <<'EOF' +## Summary + +- Adds a layered predicate to `selectMergedEntries` that renders an orphan ghost iff JSONL has stalled past the proxy (timestamp gate) AND the ghost is not sidecar-shaped (Claude Code title-gen / predict-next-prompt fingerprint backstop). +- Adds `lastJsonlEntryAt` to `SessionRuntime`, stamped from live JSONL ingest and the resume-time initial-history bootstrap. +- Bumps the orphan TTL from 3 s to 30 s — with the new predicate, orphan-ness merely qualifies a ghost as RENDER-ELIGIBLE; rules 4 and 5 still gate visibility, so the TTL semantics are now "how long before we conclude JSONL had its chance," not "how fast we paint fallback." +- Strips stale "Phase 3 will delete SemanticStreamingTurn" / "transcript-first feed with a ghost overlay" comments that described an unfinished plan; comments now match current behavior. + +## Why two rules, not one + +Production bundles show the dominant orphan-ghost source is Claude Code's auxiliary calls (predict-next-prompt, title-gen, branch-name gen) firing after the LAST real JSONL entry of a session. The timestamp predicate (`updatedAt > lastJsonlEntryAt`) cannot distinguish those from a real stuck-mid-turn ghost — both have `updatedAt` past the JSONL tail. The shape filter (single ≤200-char text block in an assistant turn) is the structural backstop. Trade-off: a real assistant turn that crashed before any JSONL write AND was a single short text block ("Done.") is also hidden. Accepted as the same trade `2a83978` made. + +## Background + +[docs/superpowers/plans/2026-05-07-ghost-system-findings.md](../tree/main/docs/superpowers/plans/2026-05-07-ghost-system-findings.md) — long-form analysis of how ghost works today, all four prior fix attempts, and why a single-rule predicate was insufficient. + +[docs/superpowers/plans/2026-05-07-ghost-rendering-predicate.md](../tree/main/docs/superpowers/plans/2026-05-07-ghost-rendering-predicate.md) — the implementation plan this PR executes. + +## Test plan + +- [x] `npm run test:ghost-fallback` (10 cases covering the predicate matrix) +- [x] `npm run build` +- [x] Manual smoke in dev: live turn renders via `SemanticStreamingTurn`, no orphan ghosts appear at the bottom of the feed within 35 s of a completed turn (title-gen / predict-next-prompt window) +- [ ] Reviewer: re-read [findings.md §10 case table](../tree/main/docs/superpowers/plans/2026-05-07-ghost-system-findings.md#10-the-actual-edge-case-ghost-was-built-for) and confirm each row is covered by the new predicate + +## Out of scope + +- atp's `mergeWithUpstream` ordered-insertion (Phase 3 of the original headless redesign). Future work; tracked in findings.md §14. +- Strengthening the proxy-side `isSidecarFlow` predicate for predict-next-prompt — independent improvement, complementary to this renderer-side filter. +- The committed-tool_result bridge in `claudeSession.ts` and its 250 ms quiet-window gate (different problem: bootstrap-replay tool_result spam). + +🤖 Generated with [Claude Code](https://claude.com/claude-code) +EOF +)" +``` + +Expected: PR URL printed. + +- [ ] **Step 4: Verify and stop** + +Print the PR URL for the user. Per `feedback_no_auto_merge.md`, do NOT merge. Hand off. + +```bash +gh pr view --json url -q .url +``` + +--- + +## Self-review checklist + +(For the writing-plans skill's own self-review pass; the human reviewer/cross-reader has their own checklist in findings.md §18.) + +- [x] **Spec coverage:** §1 (timestamp gate), §1 (shape filter), §1 (TTL bump), §1 (comment cleanup), §1 (existing-test-only) all covered by Tasks 1-7. +- [x] **No placeholders:** every code block is full content, every command is concrete. +- [x] **Type consistency:** `SessionRuntime.lastJsonlEntryAt` type matches across Task 1 (declaration), Task 2 (read + write), Task 3 (read + write), Task 5 (read). The new field is added to `emptyRuntime()` so existing callers don't break. `ghostHasSidecarShape` signature matches between Task 4 (test, indirect) and Task 5 (definition). `mergeWithUpstream` import added back in Task 5. +- [x] **Branch hygiene:** Task 0 covers worktree creation per project memory. Task 9 stops at PR open per `feedback_no_auto_merge.md`. No new test files per `feedback_no_test_bloat.md`. +- [x] **Verification:** Task 4 specifies test FAILS at HEAD (catches sequence-of-events errors). Task 5 specifies test PASSES post-implementation. Task 8 has explicit grep checks for stale references. diff --git a/docs/superpowers/plans/2026-05-07-ghost-system-findings.md b/docs/superpowers/plans/2026-05-07-ghost-system-findings.md new file mode 100644 index 000000000..e3c022d3d --- /dev/null +++ b/docs/superpowers/plans/2026-05-07-ghost-system-findings.md @@ -0,0 +1,908 @@ +# Ghost system — findings, drift, and the actual fix + +**Status:** synthesis doc, not yet executed. Written 2026-05-07 after a long +investigation triggered by orphan ghosts piling up at the bottom of the +feed across multiple Claude sessions. + +**Audience:** another agent reading this cold. They will validate the +analysis and either green-light the implementation plan in §11 or push +back. Do NOT execute anything from §11 without re-reading the code at the +file:line references in §13 — comments may have drifted again by the time +you see this. + +**TL;DR:** +- Ghost was designed to be the single live render path; that plan + ("Phase 3" in the headless redesign) never landed. +- `SemanticStreamingTurn` still owns the live current-turn render off the + semantic channel. Ghost is a parallel ledger that nothing visible + reads in the current code (`mergedEntries.ts:60-69` returns + `runtime.entries` unchanged). +- Three previous attempts to make ghost-merge rendering work + (`10e4fc5`, `686b94e`, `2a83978`) regressed each time. The common + failure: the orphan TTL fires for both "JSONL stalled past the proxy" + (the actual race case) and "sidecar leaks Claude Code never logs to + JSONL" (title-gen, predict-next-prompt). Tail-appending both produces + permanent stale rows. +- The fix is to render an orphan ghost only when its `_atp.updatedAt` is + strictly newer than the newest JSONL entry timestamp we have observed + for the session. That predicate distinguishes the two cases by + construction. +- Cleanup is small (~few hundred lines, mostly comments). No + architectural surgery. +- `crash recovery` was a fiction in earlier framings — corrected here. + +--- + +## 1. What ghost is, plainly + +Two things are watching the same conversation: + +1. **Claude Code (or Codex)** writes the *authoritative* JSONL transcript + on disk. It batches writes every 100 ms (10 ms for remote sessions + per `claude-code-src/full/utils/sessionStorage.ts:FLUSH_INTERVAL_MS`). + Codex's `RolloutRecorder` queues writes through a tokio mpsc channel + that drains on flush barriers. Either way, the file is structurally + behind the live network stream. + +2. **Our proxy** (mitmproxy with our addon) sees the model's reply + *as it streams in*. It feeds those events into + `ClaudeProxyAdapter` / `CodexResponsesAdapter`, which publish a flat + stream of semantic events (`turn_started`, `text_delta`, + `tool_input_delta`, `tool_input_finalized`, `block_completed`, + `tool_result`, `turn_completed`, etc.). + +cc-shell wants to show one feed. With both inputs available: +- *render only JSONL* → visible delay (model is talking, screen is blank). +- *render only proxy* → no durable record once cc-shell exits; on + resume the partial turn is gone. +- *render both naively* → same sentence appears twice during the lag + window. + +Ghost was supposed to be the bridge: one merged feed, with provisional +rows that swap to authoritative when JSONL catches up. + +## 2. What ghost is, technically + +Defined in `agent-transcript-parser/src/ghost.ts`. A ghost is a +`ClaudeEntry` with an `_atp` sidecar: + +```ts +type AtpGhostSidecar = { + origin: 'ghost' + turnId: string // Claude message.id / Codex response_id + blockIndex: number // 0..N within the turn + createdAt: number // ms epoch, set once + updatedAt: number // ms epoch, bumped on every snapshot + supersededBy?: string // real upstream uuid once reconciled + orphanedAt?: number // set after TTL elapses without a match + context?: Record // free-form: source/messagePhase/toolUseId +} +``` + +uuid is deterministic: `g--`. That determinism is +load-bearing — ghost logs are append-only JSONL and the reducer +(`reduceGhostLog`) picks the freshest snapshot per uuid by `updatedAt`. +No mutation, no rewrite, no lock files. Quote from +`packages/agent-transcript-parser/src/ghost.ts:71-96`: + +> WHY deterministic instead of random: +> Ghost logs are append-only JSONL. The reducer's last-write-wins rule +> only works if repeated snapshots for the same block share a key. A +> random uuid per write would force the reducer to invent its own +> grouping — a task with no non-heuristic solution for non-adjacent +> writes. +> +> The `g-` prefix ensures a ghost uuid can never collide with a Claude +> or Codex uuid in the same file. Claude uses `crypto.randomUUID()` +> which produces RFC-4122 uuids; those never start with `g-`. + +Lifecycle, from `packages/agent-transcript-parser/docs/ghost.md:30-40`: + +``` +created ──► updated* ──► superseded + \──► orphaned +``` + +Reconciliation is consumer-driven. atp ships `mergeWithUpstream` as a +reference merger; it appends unsuperseded ghosts at the **tail** of the +upstream entry list. That tail-append is what every previous render +attempt got wrong (see §8). + +Atp is intentionally library-quality: no IO, no event subscriptions, no +cc-shell vocabulary leak. + +## 3. The components in cc-shell + +| File | Role | +|---|---| +| `packages/agent-transcript-parser/src/ghost.ts` | atp primitives: `createGhost`, `updateGhost`, `supersedeGhost`, `orphanGhost`, `reduceGhostLog`, `reduceGhostLogSansSuperseded`, `mergeWithUpstream`. | +| `packages/agent-transcript-parser/src/types.ts` | `AtpGhostSidecar`, `GhostEntry`, `ClaudeContentBlock`, etc. | +| `src/main/ghostJournal.ts` | Disk writer. One file per session at `/ghost-logs/.ghost.jsonl`. 100 ms batched drain. Separate from the CLI's own JSONL — never write into Claude Code's or Codex's transcript files. | +| `src/main/ipc/ghost.ts` | IPC handlers `ghost:append` (fire-and-forget) and `ghost:read` (replay full log on resume). | +| `src/preload/api/ghost.ts` | Renderer-side bridge. `window.api.ghostAppend` / `window.api.ghostRead`. | +| `src/renderer/src/workspace/ghosts.ts` | Renderer reducer. `ghostsFromSemanticTurn` (mints/updates from semantic events), `reconcileUpstream` (supersedes on JSONL ingest), `orphanStale` (TTL sweep), `gcSupersededGhosts`, `ghostsToPersist` (diff-based persistence). | +| `src/renderer/src/workspace/mergedEntries.ts` | The render gate. `selectMergedEntries` decides which ghosts (if any) are visible in the main feed. **This is the file the fix changes.** | +| `src/renderer/src/workspace/tile-tree/TileLeaf.tsx` | Calls `selectMergedEntries(runtime, runtime.semantic.currentTurn?.turnId ?? null)` and passes the result as `entries={...}` to `Feed`. | +| `src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts` | Wires up the ghost reducer at the semantic ingest site (`ghostsFromSemanticTurn`), the JSONL ingest site (`reconcileUpstream`), the orphan sweep timer (`orphanStale` every 1 s, TTL currently 3 s), and disk persistence (`ghostsToPersist` → `ghostAppend`). | +| `src/renderer/src/workspace/hook/actions/session.ts` | On session spawn: fire-and-forget reads the ghost log, folds via `reduceGhostLogSansSuperseded`, merges into the runtime ghost map, then runs `reconcileUpstream` against any already-loaded JSONL entries. Persists supersede records produced by that pass. | +| `src/renderer/src/workspace/workspaceState.ts` | `SessionRuntime.ghosts: Map` field; initialized empty in `emptyRuntime()`. | +| `src/renderer/src/features/feed/ui/semantic/StreamingTurn.tsx` | `SemanticStreamingTurn` — the **other** live owner. Renders the current turn directly from `runtime.semantic.currentTurn` (the semantic reducer), not from ghosts. Mounted by `Feed.tsx:912-916` whenever `runtime.semantic.currentTurn !== null`. | +| `src/providers/claude/runtime/claudeSession.ts` | Sidecar opt-in (`getSessionModel`), and the committed `tool_result` bridge with its 250 ms quiet-window gate (`readyForLiveBridge`). The bridge is unrelated to ghost rendering — it deals with bootstrap-replay tool_result spam. | +| `packages/claude-code-headless/src/proxy/ClaudeProxyAdapter.ts` | The proxy-side sidecar predicate (`isSidecarFlow`) that demotes title-gen / branch-name / compaction calls to `attribution: 'secondary'` so they don't open a turn at all. Catches some sidecar leaks but not all (predict-next-prompt slips through). | + +## 4. The lifecycle wired through cc-shell + +Per session, the dance is: + +1. **Mint.** Proxy emits `turn_started` → `text_delta` (etc.). The + semantic reducer (`foldSemanticEvent`) updates + `runtime.semantic.currentTurn`. `useIpcSubscriptions.ts:711-715` + calls `ghostsFromSemanticTurn(currentTurn, sessionId, current.ghosts)`. + That returns the previous map unchanged for no-op events + (reference-stable, see `ghosts.ts:235-242`) or a new map with one + `createGhost` / `updateGhost` per affected block. + +2. **Persist.** `ghostsToPersist(prev, next)` returns the entries whose + `updatedAt` changed (or are new). Each is sent over IPC via + `window.api.ghostAppend(sessionId, ghost)`. Main batches at 100 ms. + This happens both on the semantic ingest path (line ~723) and on the + JSONL ingest path (line ~1133, capturing supersede records). + +3. **Reconcile.** When JSONL entries land via `onSessionJsonlEntries`, + `reconcileUpstream(entry, ghosts)` walks the ghost map and + supersedes any whose: + - `turnId === entry.message.id` (Claude rule) + - OR `turnId === entry.codexTurnId` (Codex rule, after + `stampCodexTurnId` ran in the rollout mapper) + - OR `_atp.context.toolUseId` / `context.callId` matches a + `tool_use` block id in the entry (provider-shared fallback) + +4. **Orphan.** `useIpcSubscriptions.ts:210-241` runs `orphanStale` every + 1 s. Any unsuperseded ghost whose `updatedAt + GHOST_ORPHAN_TTL_MS < + now` gets `orphanedAt` set. `GHOST_ORPHAN_TTL_MS = 3000` currently. + +5. **GC.** `gcSupersededGhosts` evicts entries that have been + superseded for `gcMs` to keep the map bounded. Called from the same + periodic tick. + +6. **Resume.** On `spawnSession` (`session.ts:212-266`), after a 0-tick + defer, read `/ghost-logs/.ghost.jsonl`, fold via + `reduceGhostLogSansSuperseded` (drops anything already superseded on + disk), merge into `runtime.ghosts` (skipping uuids the live session + already produced), and re-run `reconcileUpstream` against + `current.entries` (which by now contains the JSONL tail loaded by + `loadInitialHistoryForSession`). Persist any newly-produced supersede + records. + +This whole dance is well-engineered and works. The problem is it ends +in step 7 — render — that no longer happens. + +## 5. Reference-stability invariants that keep the system fast + +Comments throughout `ghosts.ts`, `useIpcSubscriptions.ts`, and the atp +package make a load-bearing claim: each reducer returns `prev` unchanged +on no-op so React memoization holds. Quote from +`ghosts.ts:235-242`: + +> WHY lazy clone: this runs on every semantic reducer tick, including +> no-op ticks (usage_updated, redundant block_started). The pre-fix +> version always allocated `new Map(prev)` at the top, which made +> `nextGhosts !== current.ghosts` always true downstream, forcing a +> setRuntimes cascade that busted every useMemo([entries]) in Feed via +> selectMergedEntries. + +Same pattern in `reconcileUpstream` (line ~352-359), `orphanStale` +(line ~451), and `foldSemanticEvent` itself (no-op short-circuit at the +end, ~lines 798-820). + +Implication for our fix: the new predicate must keep this contract. +Specifically, `selectMergedEntries` should return `runtime.entries` by +identity when no ghost survives the predicate, NOT a fresh +`[...entries]`. The pre-fix `mergeWithUpstream` returned a fresh array +even when `trailing` was empty, busting Feed memos. + +## 6. The dual-owner problem (the architectural rot) + +`Feed.tsx:912-916` mounts `SemanticStreamingTurn` whenever +`runtime.semantic.currentTurn !== null`. That component renders blocks +from `currentTurn.blocks` directly — not from ghosts. + +So during a normal turn: +- semantic events arrive → `currentTurn` updates → `SemanticStreamingTurn` + re-renders with fresh block content. +- *In parallel*, the same events mint/update ghost entries. +- JSONL lands → `currentTurn` closes (`turn_completed` → + `currentTurn = null` if no pending tool_result), `SemanticStreamingTurn` + unmounts, JSONL row takes over. +- *In parallel*, `reconcileUpstream` supersedes the ghost. + +Two systems doing the same job in parallel. The ghost system was +supposed to *replace* `SemanticStreamingTurn` — that's the literal +content of "Phase 3" referenced everywhere in the comments. From +`mergedEntries.ts:71-92`: + +> Phase 3 of the original headless redesign will delete +> SemanticStreamingTurn entirely and render everything through ghosts + +> the merged feed. Until then, single-ownership in the main feed is the +> surgical fix. + +That deletion never happened. `SemanticStreamingTurn` is fine and +serves the live render need correctly. So ghost-as-render became +redundant — except for one residual case where ghost is the only +thing that knows: **when JSONL has stopped writing past the proxy.** + +## 7. Comment drift to be aware of + +The codebase has multiple WHY comments that describe ghost as the +canonical live render path. They're all from the period when Phase 3 +was actively planned. Examples: + +- `ghosts.ts:1-40` block comment: "The fix is a transcript-first feed + with a ghost overlay." — describes the design intent that wasn't + finished. +- `Feed.tsx:170-180`: "duplicate class is now prevented at its source" + via the ghost reducer. Half-true: ghost reconciliation is doing work, + but the duplicate-render avoidance currently relies on + `SemanticStreamingTurn` being the sole owner of the live turn AND + ghosts being hidden in the main feed. +- `mergedEntries.ts` various: described as a wrapper around + `mergeWithUpstream`. Currently it isn't — it returns `entries`. +- `mergedEntries.ts:71-92`: describes the "Phase 3 delete" plan as if + it's the next item on the queue. It hasn't moved in months. +- `TileLeaf.tsx:339-369`: call-site comments still say "earlier turns' + orphaned or still-unreconciled ghosts fall through" — they don't. + +These don't lie about facts that ARE in the code (the ghost reducer is +real, `mergeWithUpstream` does what it says) but they overstate what +the *render path* currently does. Future reader trying to understand +why a ghost isn't showing will be misled. + +## 8. The fix history (every previous attempt) + +### `10e4fc5` (2026-04-20) — "hide non-current ghosts from the main transcript" + +Origin: session `69e61aa3-38af-428c-9f31-a614a6c4c4a7` showed +`entry:g-019dab01-...` (older ghost answer) rendering BELOW +`semantic:019dab02-...` (newer turn). atp's tail-append put a stale +old-turn ghost below newer committed rows. + +Fix: filter all non-current-turn ghosts out of the merge. Render the +current turn via `SemanticStreamingTurn`. Result: stale-bottom-row bug +gone, but no orphan rendering at all. + +Quote from the commit message: + +> This is an immediate UI sanity fix, not the final architecture. It +> stops the transcript from inverting conversational order. + +### `686b94e` (2026-04-24) — "render orphaned ghost fallback entries" + +Came back to add the orphan-fallback story: orphaned ghosts (TTL +fired) ARE the only record of what proxy saw for that turn, so render +them. Code added the orphan-only filter and re-enabled +`mergeWithUpstream` for those. + +Failure mode: didn't distinguish the two reasons a ghost orphans. +Sidecar leaks (title-gen, predict-next-prompt) orphan because Claude +Code never logs them to JSONL — same TTL signal as a stuck-mid-turn +case. Result: 7+ short fragments parked permanently at the bottom of +sessions. + +### `2a83978` (2026-05-07) — "suppress orphan ghosts with title-gen / predict-next-prompt shape" + +Symptom fix on top of `686b94e`. Added a renderer-side shape filter: +orphan ghost is suppressed if it's an assistant message with a single +text block ≤ 200 chars. Rationale from the commit message: + +> Forensic confirmation from debug bundle 2026-05-07T08-26-35-212-5d948ab5: +> 7 orphan ghosts visible at the bottom of the feed, none of 23 flows +> demoted, all sidecar turns 12-41 chars and 188-602ms — well below any +> real assistant turn (≥76 chars, ≥808ms in the same bundle). + +Failure mode: the 200-char cap is empirical against ONE bundle. A +sidecar variant that produces longer text (predict-next-prompt with +fuller context) defeats it. Genuine short assistant turns ("Done.", +"OK") would also be suppressed, accepted as a trade-off. Tool_use +ghosts not covered at all. + +### Current branch `fix/hide-orphan-ghost-tail` (2026-05-07, this session) + +User saw orphan ghosts from May 2026 sessions still leaking through +the shape filter. Initial branch went the conservative route again: +hide ALL ghosts from the main feed. Same as `10e4fc5`'s shape but +permanent. + +This is what the branch currently does. It works (no stale rows) but +also kills the legitimate stuck-mid-turn case. After the user +hammered on this, we landed on the fix described in §10. + +## 9. Why all four attempts had the same shape + +Each attempt asked: "is this ghost stale enough to render?" The TTL +fires identically for: + +- (a) JSONL is healthy and writing other turns, but this *specific* + proxy event was a sidecar Claude Code never writes (title-gen, + predict-next-prompt, branch-name gen, compaction summary, hook + agent). +- (b) JSONL has actually stopped writing for this session entirely, + while proxy is still emitting events for an in-flight turn. + +A TTL-based predicate cannot distinguish these. (a) is the dominant +population in production bundles; (b) is rare but is the only case +ghost rendering should fire on. Tail-appending (a) produces the bug we +keep re-introducing. + +The proxy-side `isSidecarFlow` predicate +(`ClaudeProxyAdapter.ts:1542-1610`) catches most of (a) at source. +It demotes flows when: +- the flow's model matches `/haiku/i` AND the session model doesn't, OR +- compound budget signal: `max_tokens ≤ 1024 AND messageCount ≤ 3`, OR +- system-prompt prefix matches one of `SIDECAR_SYSTEM_PROMPT_PREFIXES` + (4 entries, English literal prefixes) + +What it misses, per the `2a83978` commit message: + +> The body-shape predicate misses the predict-next-prompt feature +> because those calls include the full conversation history +> (messageCount > 3 so the compound budget signal fails) and use a +> system prompt not in the 4-entry SIDECAR_SYSTEM_PROMPT_PREFIXES list. + +So even with the proxy-side filter doing its job for the catchable +cases, some sidecar leaks reach the renderer. The renderer-side fix +must handle them. + +## 10. The actual edge case ghost was built for + +User's framing, restated: + +> when we get in that weird stale state where like 50 seconds goes by a +> restart and claude and codex stops writing to the jsonl before the +> WHOLE turn is finished, the 100 milliseconds is just nothing? in that +> case we just use the semantic channel until the jsonl channel +> replaces it. BUT if we see that the semantic channel is growing with +> no updates in jsonl that is the race case ghosts where meant to solve + +That's exactly right. The signal we want is per-session, not per-ghost: +*has JSONL gone silent while proxy is still alive (or has it stopped +permanently as of this resume)?* + +The cleanest implementation of that signal is to compare against +**the newest JSONL entry timestamp this session has observed**. Then a +ghost is render-eligible iff its `_atp.updatedAt` is strictly newer +than that timestamp. + +Walking the cases: + +| Scenario | `lastJsonlEntryAt` | ghost `updatedAt` | predicate | correct? | +|---|---|---|---|---| +| Normal turn streaming, JSONL 100 ms behind | t-0.1 | t-0 | hidden (orphan TTL hasn't fired) | ✓ — `SemanticStreamingTurn` renders | +| Normal turn done, JSONL caught up | t | t (then superseded) | hidden (superseded) | ✓ | +| Sidecar leak (title-gen) at t=5, JSONL kept writing other turns to t=100 | t=100 | t=5 | hidden | ✓ | +| JSONL stopped at t=22, proxy still emitting, ghost `updatedAt=27` | t=22 | t=27 | rendered | ✓ — this is the case ghost was built for | +| Resumed after crash; JSONL tail loaded ends at t=22; ghost log has t=27 ghost reconciled to no-match | t=22 | t=27 | rendered | ✓ — the partial-turn-from-previous-run is recovered | +| Resumed; old sidecar at t=50 in ghost log, JSONL has entries up to t=100 | t=100 | t=50 | hidden | ✓ | + +The predicate works for both live operation and resume because both +sides are **wall-clock-when-the-event-was-observed**, comparable. + +## 11. The implementation plan + +### 11.1. Data model + +Add to `SessionRuntime` in +`src/renderer/src/workspace/workspaceState.ts` (next to `ghosts`): + +```ts +/** Wall-clock ms of the newest JSONL entry's `timestamp` field we + * have observed for this session. Compared against ghost + * `_atp.updatedAt` in `selectMergedEntries`: when a ghost's update + * is strictly newer than this value AND the ghost is orphaned, + * JSONL has stalled past the proxy and the ghost is the only record + * of what happened — render it. When the ghost is older than this + * value, JSONL kept writing past it (sidecar leak that Claude Code + * was never going to log) — hide it. Null until any JSONL entry + * has been observed. Reset to null on session removal. */ +lastJsonlEntryAt: number | null +``` + +Initialize `null` in `emptyRuntime()`. + +### 11.2. Stamp from live JSONL ingest + +In `src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts`, +inside the `onSessionJsonlEntries` setRuntimes block (~line 1170 area +where the runtime patch is built), walk `appended` and update the +field: + +```ts +let lastJsonlEntryAt = current.lastJsonlEntryAt +for (const entry of appended) { + const ts = (entry as { timestamp?: unknown }).timestamp + if (typeof ts === 'string') { + const ms = Date.parse(ts) + if (Number.isFinite(ms) && (lastJsonlEntryAt === null || ms > lastJsonlEntryAt)) { + lastJsonlEntryAt = ms + } + } +} +``` + +Add `lastJsonlEntryAt` to the runtime spread (line ~1175). Use +`entry.timestamp` (ISO string, present on both Claude entries and +Codex-mapped entries) NOT `Date.now()` so the comparison is +apples-to-apples with ghost `_atp.updatedAt`. + +NEEDS VERIFICATION: confirm that mapped Codex entries via +`mapCodexRolloutToFeedEntries` consistently set `timestamp`. Looking +at `src/renderer/src/workspace/codex/rollout.ts`, +`codexConversationEntryFromMessageItem` accepts `timestamp` as +parameter and writes it into the entry. Source value comes from the +rollout entry's outer `timestamp` field (Codex JSONL has it on every +line). Should be consistently populated. Same path for tool_use / +tool_result mapped entries. **Validator: grep +`src/renderer/src/workspace/codex/rollout.ts` and confirm every emit +path threads `timestamp` through.** + +### 11.3. Stamp from initial-history load + +`src/renderer/src/workspace/hook/actions/initialHistory.ts` (~line 193 +where `entries:` is set): + +```ts +let lastJsonlEntryAt = current.lastJsonlEntryAt +for (const entry of initialEntries) { + const ts = (entry as { timestamp?: unknown }).timestamp + if (typeof ts === 'string') { + const ms = Date.parse(ts) + if (Number.isFinite(ms) && (lastJsonlEntryAt === null || ms > lastJsonlEntryAt)) { + lastJsonlEntryAt = ms + } + } +} +``` + +`history.ts` (older-history pagination, line ~178): does NOT touch +this field. Prepended entries are by construction older than entries +already in `runtime.entries`. + +### 11.4. The new `selectMergedEntries` + +`src/renderer/src/workspace/mergedEntries.ts` — replace the body: + +```ts +import type { Entry } from '@shared/types/transcript' +import type { SessionRuntime } from '@renderer/workspace/workspaceState' +import type { GhostEntry } from 'agent-transcript-parser/ghost' +import { mergeWithUpstream } from 'agent-transcript-parser/ghost' + +/** + * Decide which (if any) ghost entries get merged into the rendered + * feed. The ghost system mints provisional records for every semantic + * event. Most of those never need to render: the live current turn is + * owned by `SemanticStreamingTurn` directly off the semantic channel, + * and JSONL catches up within ~100 ms via `reconcileUpstream`, which + * supersedes the ghost. + * + * The one case ghost rendering exists for is JSONL stalling past the + * proxy. Two situations produce that: + * 1. Live: agent process gets wedged, JSONL stops writing while + * proxy keeps emitting events for the in-flight turn. + * 2. Resume after crash: ghost log on disk has events past the + * newest JSONL entry; JSONL never caught up before the previous + * run died. + * + * Both are detected by the same predicate: a ghost is render-eligible + * iff its `_atp.updatedAt` is strictly newer than the newest JSONL + * entry timestamp we have observed for this session + * (`runtime.lastJsonlEntryAt`). Sidecar leaks (Claude Code routing + * title-gen / predict-next-prompt / branch-name gen calls through + * /v1/messages without writing to the rollout) fail this check by + * construction, because JSONL kept writing real turns past them. + * + * Full rule: + * 1. Skip if superseded. + * 2. Skip if not yet orphaned — JSONL might still arrive within TTL. + * 3. Skip if `turnId === currentTurnId` — `SemanticStreamingTurn` + * owns the live turn render. Surfacing a ghost for the same turn + * would double-render. + * 4. Skip if `_atp.updatedAt <= lastJsonlEntryAt` — JSONL has + * written past this point, so the ghost is either already + * reconciled or a sidecar leak. + * + * Surviving ghosts are tail-appended via atp's `mergeWithUpstream` + * with `trustSupersededFlag: true`. Tail-append is correct here + * because by predicate-3 these ghosts are NOT for the active turn, + * and by predicate-4 they are newer than every committed entry — + * so they belong at the very end chronologically. + * + * Reference stability: when no ghost survives the predicate, return + * `runtime.entries` by identity (NOT a fresh array). Feed's row + * memos rely on entries-array identity to skip re-renders; + * `[...entries]` would bust them on every tick. + */ +export function selectMergedEntries( + runtime: SessionRuntime, + currentTurnId: string | null, +): Entry[] { + const { ghosts, entries, lastJsonlEntryAt } = runtime + if (ghosts.size === 0) return entries + + const visible = new Map() + for (const [uuid, ghost] of ghosts) { + if (ghost._atp.supersededBy !== undefined) continue + if (ghost._atp.orphanedAt === undefined) continue + if (currentTurnId !== null && ghost._atp.turnId === currentTurnId) continue + if (lastJsonlEntryAt !== null && ghost._atp.updatedAt <= lastJsonlEntryAt) continue + visible.set(uuid, ghost) + } + if (visible.size === 0) return entries + + return mergeWithUpstream(entries, visible, { + trustSupersededFlag: true, + }) as Entry[] +} + +export function shouldShowSemanticStreaming(runtime: SessionRuntime): boolean { + return runtime.semantic.currentTurn !== null +} +``` + +The 200-char sidecar shape filter from `2a83978` is **deleted**. The +new predicate replaces it correctly: a sidecar's `updatedAt` is by +construction older than the next real-turn JSONL entry, so rule 4 +catches it. The shape filter was a heuristic; the new rule is structural. + +### 11.5. Bump the orphan TTL + +`useIpcSubscriptions.ts:97`: + +```ts +const GHOST_ORPHAN_TTL_MS = 30000 // was 3000 +``` + +3 s was set when `686b94e` wanted aggressive fallback rendering. Now +that orphan-ness gates rendering AND we have an additional newer-than- +JSONL check, 3 s is too aggressive — a slow Read of a 5000-line file +can take a few seconds during which the proxy emits no new events, +which would currently flip the ghost to orphaned mid-turn. 30 s +matches atp's library default and safely covers all known +slow-tool_result cases without prematurely qualifying a ghost. + +Leave `GHOST_ORPHAN_SWEEP_MS = 1000` alone — that's the polling rate. +The 1 s sweep is what drives the visibility flip when JSONL goes +silent live: the orphan flag flips → setRuntimes fires → re-render → +predicate re-evaluates → ghost appears. + +### 11.6. Comment cleanup + +Update these block comments to match reality. Each rewrite is small. + +- `mergedEntries.ts` file-level comment: replace the "thin wrapper + around `mergeWithUpstream`" / "Phase 3 will delete + `SemanticStreamingTurn`" framing with: ghost rendering is reserved + for the JSONL-stalled-past-proxy case; otherwise the live turn is + owned by `SemanticStreamingTurn` and JSONL owns committed history. +- `ghosts.ts:1-40` block comment: rewrite. New framing: "Ghost is a + parallel disk-backed ledger of semantic events. Used for (a) + reconciliation against JSONL (live duplicate-render avoidance), (b) + the JSONL-stalled-past-proxy case where ghost is the only record of + what happened — see `selectMergedEntries`. Most ticks the ghost map + has no rendered output; the persistence and reconciliation work + goes on in the background." +- `TileLeaf.tsx:333-369` call-site comments: tighten to the new + rule. Drop the "earlier turns' orphaned or still-unreconciled + ghosts fall through" claim — they don't unless predicate 4 is + satisfied. +- `Feed.tsx:170-180`, `:670-676`, `:945-950`: leave alone. They + describe `SemanticStreamingTurn` ownership accurately. +- `mergedEntries.ts` `shouldShowSemanticStreaming` doc-comment: trim. + The body is one line; the docstring doesn't need to be 20. + +DO NOT touch: +- atp ghost primitives (`packages/agent-transcript-parser/`) +- `ghostJournal.ts`, IPC, preload — persistence layer is fine. +- `reconcileUpstream`, `orphanStale`, `gcSupersededGhosts`, + `ghostsFromSemanticTurn`, `ghostsToPersist`. +- `claudeSession.ts` committed-tool_result bridge — different problem. +- `ClaudeProxyAdapter.isSidecarFlow` and friends — keep proxy-side + filtering. Catching sidecars at the proxy is still preferred when + possible; the renderer predicate is the safety net. + +### 11.7. Test update + +`scripts/test-ghost-fallback.ts` already exists and is wired to the +`test:ghost-fallback` script in `package.json`. **Do not add new test +files** (per project memory `feedback_no_test_bloat.md`). Modify the +existing one. + +Replace the current "all ghosts hidden" assertions with the four-rule +matrix: + +```ts +// 1. orphaned + updatedAt > lastJsonlEntryAt + no current turn → render +// 2. orphaned + updatedAt < lastJsonlEntryAt → hide (sidecar/catchup case) +// 3. orphaned + turnId === currentTurnId → hide (live owner is SemanticStreamingTurn) +// 4. not orphaned → hide (handoff window, JSONL might still land) +// 5. superseded → hide +``` + +The existing helper `emptyRuntime()` will pick up the new +`lastJsonlEntryAt: null` automatically. Tests set it via direct field +assignment on the test runtime, same pattern as `runtime.entries =` +in the current test. + +### 11.8. Branch hygiene + +Current branch `fix/hide-orphan-ghost-tail` describes the wrong +behavior now. Rename to something like +`fix/render-ghost-when-jsonl-stalls` or +`fix/ghost-rendering-predicate`. Force-push fine — branch isn't +shared upstream (gh remote is on `Juliusolsson05`, see +`reference_gh_account.md`). The current single commit on this branch +will be replaced by a more accurate commit message. + +Per project memory `feedback_worktree_default.md`, branch work goes +in `.worktrees/` with main checkout staying on main. Verify +this branch is in a worktree before continuing. + +## 12. Open questions / things to verify before committing + +1. **Codex `entry.timestamp` coverage.** Verify every emit path in + `src/renderer/src/workspace/codex/rollout.ts` populates + `timestamp`. If any path drops it, `lastJsonlEntryAt` won't + advance for those bursts and orphan ghosts could erroneously + render. Mitigation if found: fall back to `Date.now()` when the + entry has no timestamp (less semantic but doesn't break the + predicate). + +2. **Compact-boundary entries.** These have a `uuid` but may not + carry `timestamp`. They're not conversation entries and shouldn't + reset the JSONL clock, but they currently flow through + `appended`. Test: confirm parse failure on missing/non-string + timestamp degrades to "no update" not "set to NaN." + +3. **First-turn fresh session.** No JSONL has been received yet + (`lastJsonlEntryAt === null`). A ghost minted from the very first + semantic event has `updatedAt > null` (treats null as 0), so + predicate 4 passes. Predicate 3 (`turnId === currentTurnId`) hides + it because `currentTurn` is open. ✓ — but ASSUMPTION: the + semantic reducer always opens `currentTurn` BEFORE the first + ghost is minted. Verify by tracing `foldSemanticEvent` for + `turn_started` (opens currentTurn) vs the first `text_delta` / + `block_started` (which is what mints the first ghost). Looking at + `foldEvent.ts:111-172`, `turn_started` runs before any block + events; `useIpcSubscriptions.ts:711-715` runs `ghostsFromSemanticTurn` + AFTER `foldSemanticEvent` AFTER the new event has updated state. + ✓ ordering is fine. + +4. **`turn_completed` / `turn_stopped` race.** When the semantic + reducer closes `currentTurn` (sets it to null) but the JSONL + entry hasn't landed yet — a window of ~100 ms where: + - predicate 1 ✗ (not yet superseded) + - predicate 2: depends on TTL — orphan_TTL=30s, so the ghost is + not orphaned yet at this moment. Predicate 2 hides it. + - predicate 3: passes (currentTurn=null, no match) + - predicate 4: passes (updatedAt > lastJsonlEntryAt by definition, + since the proxy emitted past the last JSONL entry) + + Predicate 2 is what saves us. The TTL must be long enough that + normal `turn_completed → JSONL lands` never crosses it. 30 s is + plenty (turn_completed → JSONL is bounded by Claude Code's 100 ms + batch; Codex by its mpsc flush, both well under 30 s in healthy + operation). + +5. **What if Claude Code's JSONL writer is healthy but lags by 5+ + seconds?** Pathological case: a really big tool_result floods + the queue, batched write delays, ghost gets to TTL=30s, predicates + 1-3 pass, predicate 4 passes (last JSONL was the user message + from a minute ago). Ghost renders. Then JSONL lands, supersedes, + ghost disappears. User sees a brief flicker. **Probability low, + visible glitch minor, acceptable.** If this becomes a problem, + tune TTL upward. + +6. **Resume + ghost log fully empty.** Edge case where the previous + session crashed before any ghost was persisted but JSONL has + partial state. Nothing to render. Predicate trivially passes for + no ghosts. ✓ — but worth a sanity test. + +7. **`gcSupersededGhosts` interaction.** Superseded ghosts get + evicted after `gcMs`. After eviction, predicate 1 has nothing to + match (the ghost is gone). Doesn't affect rendering — the ghost + was already hidden. ✓ + +8. **Multiple sessions, one stuck.** `lastJsonlEntryAt` is + per-session; one stuck session does not affect another's + predicate. ✓ + +9. **`bootstrapping` flag.** Feed suspends auto-scroll while + `bootstrapping=true`. Ghost rendering during bootstrap could + cause a small jump when the orphan flips to visible. Probably + fine because the orphan TTL (30 s) is much longer than the + bootstrap quiet window (150 ms). + +## 13. File-and-line cheat sheet + +For the next agent. All paths are absolute under +`/Users/juliusolsson/Desktop/Development/cc-shell/`. + +| Concern | File | Line | +|---|---|---| +| `SessionRuntime` shape | `src/renderer/src/workspace/workspaceState.ts` | 248-394 | +| `emptyRuntime()` | `src/renderer/src/workspace/workspaceState.ts` | 425-481 | +| `ghosts` field doc | `src/renderer/src/workspace/workspaceState.ts` | 379-393 | +| `ghostsFromSemanticTurn` | `src/renderer/src/workspace/ghosts.ts` | 230-287 | +| `reconcileUpstream` | `src/renderer/src/workspace/ghosts.ts` | 348-428 | +| `orphanStale` | `src/renderer/src/workspace/ghosts.ts` | 447-462 | +| `gcSupersededGhosts` | `src/renderer/src/workspace/ghosts.ts` | 478-493 | +| `ghostsToPersist` | `src/renderer/src/workspace/ghosts.ts` | 513-528 | +| Live JSONL ingest (`onSessionJsonlEntries`) | `src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts` | 846-1232 | +| Ghost reconcile call | `src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts` | 1111-1135 | +| Runtime patch (entries:) | `src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts` | 1170-1210 | +| Orphan sweep | `src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts` | 210-241 | +| TTL constant | `src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts` | 97 | +| Ghost mint on semantic tick | `src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts` | 695-725 | +| Initial history load | `src/renderer/src/workspace/hook/actions/initialHistory.ts` | 102-200 | +| Older-history pagination | `src/renderer/src/workspace/hook/actions/history.ts` | 170-200 | +| Resume ghost bootstrap | `src/renderer/src/workspace/hook/actions/session.ts` | 199-266 | +| `selectMergedEntries` (current) | `src/renderer/src/workspace/mergedEntries.ts` | full file | +| `TileLeaf` call-site | `src/renderer/src/workspace/tile-tree/TileLeaf.tsx` | 333-379 | +| `SemanticStreamingTurn` mount | `src/renderer/src/features/feed/ui/Feed.tsx` | 912-916 | +| `SemanticStreamingTurn` impl | `src/renderer/src/features/feed/ui/semantic/StreamingTurn.tsx` | full file | +| Semantic reducer | `src/renderer/src/workspace/semantic/foldEvent.ts` | full file | +| `currentTurn` lifecycle | `src/renderer/src/workspace/semantic/foldEvent.ts` | 111-172, 707-731 | +| Codex rollout mapper | `src/renderer/src/workspace/codex/rollout.ts` | full file | +| `stampCodexTurnId` | `src/renderer/src/workspace/codex/rollout.ts` | 198-201 | +| Claude proxy session | `src/providers/claude/runtime/claudeSession.ts` | full file | +| Committed tool_result bridge | `src/providers/claude/runtime/claudeSession.ts` | 378-459 | +| Sidecar opt-in (getSessionModel) | `src/providers/claude/runtime/claudeSession.ts` | 257-292 | +| Ghost journal writer | `src/main/ghostJournal.ts` | full file | +| Ghost IPC handlers | `src/main/ipc/ghost.ts` | full file | +| Ghost preload bridge | `src/preload/api/ghost.ts` | full file | +| atp ghost primitives | `packages/agent-transcript-parser/src/ghost.ts` | full file | +| atp ghost types | `packages/agent-transcript-parser/src/types.ts` | 38-105 | +| atp `mergeWithUpstream` | `packages/agent-transcript-parser/src/ghost.ts` | 410-451 | +| atp `MergeOptions` | `packages/agent-transcript-parser/src/ghost.ts` | 354-381 | +| Sidecar predicate | `packages/claude-code-headless/src/proxy/ClaudeProxyAdapter.ts` | 1542-1610 | +| Sidecar prefix list | `packages/claude-code-headless/src/proxy/ClaudeProxyAdapter.ts` | 385-390 | +| `MAX_TOKENS_SIDECAR_THRESHOLD` | `packages/claude-code-headless/src/proxy/ClaudeProxyAdapter.ts` | 410 | +| `AUXILIARY_MESSAGE_COUNT_THRESHOLD` | `packages/claude-code-headless/src/proxy/ClaudeProxyAdapter.ts` | 429 | + +## 14. Alternative approaches we considered and rejected + +### Option A — rip the ghost system out + +Delete `ghosts.ts`, `ghostJournal.ts`, IPC, preload, the field on +`SessionRuntime`, the orphan sweep, the bootstrap merge, the +reconcile-on-ingest. Net ~500 lines deleted. + +Rejected because: +- Closes the door on Phase 3 cleanly handling the + JSONL-stalled-past-proxy case. The user cared about that case. +- Loses the disk-backed forensic record that debug bundles benefit + from. +- The work needed to bring it back would be larger than the work to + fix the predicate. + +### Option B — keep plumbing, gate behind debug flag + +Same renderer behavior as today (ghosts hidden) but only mint / +persist when feed-debug is open or `CC_SHELL_GHOST_DEBUG=1` is set. + +Rejected because: +- Doesn't actually fix the bug (the user-visible regression on stuck + mid-turn cases). Just makes the system smaller. +- Adds a runtime mode bit. More state to reason about. + +### Option C — Phase 3 (delete `SemanticStreamingTurn`, render via ordered ghost insertion) + +The original architectural plan. Add ordered insertion to atp +`mergeWithUpstream` (anchor by `parentUuid` / `turnId` / nearest +committed neighbor). Delete `SemanticStreamingTurn`. Render the live +turn entirely through the merged feed. + +Rejected as the immediate fix because: +- It's a project, not a cleanup. Several days of work in atp + cc-shell. +- Ordered insertion has its own correctness traps (parent uuid chains + break on resume; turn_id ordering depends on rollout invariants + that vary between providers). +- Doesn't deliver new user value, only refactor value. +- Future Phase 3 still benefits from the smaller predicate landing + first as a stepping stone. + +### Option D — chosen — predicate fix in `selectMergedEntries` + +Smallest viable fix. Captures the exact case ghost was built for. +Sidecar leaks structurally excluded by the timestamp comparison. +Few hundred lines, mostly comments. See §11. + +## 15. What is NOT being changed + +For the next agent's benefit, things that LOOK related but are out +of scope: + +- The proxy-side sidecar predicate (`ClaudeProxyAdapter.isSidecarFlow`). + Catches the catchable cases at source. Keep it. Extending the + prefix list and budget thresholds is independent work; this fix + doesn't depend on it. +- The committed `tool_result` bridge in `claudeSession.ts`. Solves + bootstrap-replay tool_result spam. Unrelated to ghost rendering. +- The orphan sweep cadence (`GHOST_ORPHAN_SWEEP_MS = 1000`). + Polling rate; the threshold is `GHOST_ORPHAN_TTL_MS`. +- atp's library code. The primitives are correct as-is. We don't + need a new merge option or a new lifecycle state. +- `gcSupersededGhosts`. Memory hygiene; doesn't affect rendering. +- The bootstrap quiet-window flag (`bootstrapping`) and its 150 ms + debounce. Unrelated; affects auto-scroll, not ghost visibility. +- The work indicator stream phase (`streamPhase`). Shows "agent is + thinking / using a tool"; not coupled to ghost. + +## 16. Useful evidence locations + +For verifying behavior against real bundles: + +- `~/.config/cc-shell/feed-debug/.jsonl` — per-session + append-only feed-debug log. Contains layered entries: + - `layer: SEM` — semantic events + - `layer: JSONL` — JSONL ingest bursts + - `layer: STATE` — bootstrap_complete, ghost_orphan_sweep, + conditions, etc. + - `layer: RENDER` — Feed's visible-row reconciliation +- `~/Library/Application Support/cc-shell/ghost-logs/.ghost.jsonl` — + the ghost log itself. Each line is a `GhostEntry` snapshot. +- `~/.config/cc-shell/debug-bundles/` — autosaved bundles with + cross-cut state (runtime summary, semantic log slice, feed-debug + slice, ghost map size, conditions). + +The session referenced in `10e4fc5` is +`69e61aa3-38af-428c-9f31-a614a6c4c4a7`. The session referenced in +`2a83978` is bundle `2026-05-07T08-26-35-212-5d948ab5`. + +## 17. What ghost is NOT (corrections to earlier framings in this thread) + +- **Not crash recovery.** Earlier in the investigation I described + ghost as "crash recovery." That was wishful — the on-disk ghost + log only contributes to user-visible state if ghosts render, and + they currently don't. The persistence is real but its only consumers + today are the bootstrap merge (which feeds back into a hidden map) + and debug bundle counters. Calling that "recovery" overstates it. + The fix in §11 *makes* it crash recovery for the stuck-mid-turn case. +- **Not the live render path.** The semantic channel via + `SemanticStreamingTurn` is. Ghost minting happens in parallel; nothing + visible reads the ghost map during normal operation. +- **Not solving the 100 ms JSONL lag.** That's solved by + `SemanticStreamingTurn` directly. Ghost is for the abnormal + long-stall case. +- **Not "the merged feed."** That term shows up in old comments. The + current feed is committed-entries-only, with `SemanticStreamingTurn` + appended for the live turn. After this fix, "merged" partially + comes back, but only for orphans past JSONL. + +## 18. Summary for the cross-reading agent + +Cross-read priorities: + +1. Verify §10's case table by walking each scenario mentally against + the four-rule predicate in §11.4. Look for missed cases. +2. Verify §12 open questions, especially Codex `timestamp` coverage + in `rollout.ts` and the `bootstrapping`/auto-scroll interaction. +3. Look for stale comments not enumerated in §7 — there are likely + more across the workspaceStore, claudeSession, and Feed paths + that drift from current behavior. +4. Sanity-check the §11.5 TTL bump (3000 → 30000) against any other + consumer of `GHOST_ORPHAN_TTL_MS`. Currently only the sweep uses + it, but verify with a grep. +5. Confirm `entry.timestamp` is consistently ISO-8601-string for both + providers. Specifically: Claude entries from + `claudeHistoryMarker`, `extractEmbeddedClaudeProgressEntry`, and + the live JSONL tail; Codex entries from + `mapCodexRolloutToFeedEntries` and its synthetic + compact-boundary / compact-summary helpers. +6. Confirm renaming `fix/hide-orphan-ghost-tail` to a more accurate + branch name does not orphan any open PR or in-flight comment. +7. Run `npm run test:ghost-fallback` after the change. The test must + still pass; if it doesn't, the predicate or test update is wrong. +8. Run `npm run build` after the change. TypeScript will catch any + field-shape mismatch on `SessionRuntime`. + +If §11 holds up, hand it back to the user for execution. From 3b04be440836e2b0371d53cec831850d02dd262a Mon Sep 17 00:00:00 2001 From: Julius Olsson Date: Thu, 7 May 2026 15:49:45 +0200 Subject: [PATCH 2/6] runtime: plumb lastJsonlEntryAt through ingest paths MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Foundation for the ghost-rendering predicate. Adds SessionRuntime.lastJsonlEntryAt and stamps it from both ingest paths: * useIpcSubscriptions.onSessionJsonlEntries — walks the burst, parses entry.timestamp ISO strings, takes the max. Includes the field in the no-change short-circuit so a burst that ONLY advances the cursor still lands. * initialHistory bootstrap — same scan applied to the loaded JSONL tail on resume, so the predicate has the right anchor against ghost _atp.updatedAt loaded from disk. history.ts (older-history pagination) deliberately does NOT touch the field — its prepended entries are by construction older than runtime.entries. No visible behavior change yet: nothing reads lastJsonlEntryAt until selectMergedEntries is rewired in the next commit. See docs/superpowers/plans/2026-05-07-ghost-system-findings.md for the full design rationale. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../workspace/hook/actions/initialHistory.ts | 21 +++++++++++ .../workspace/hook/ipc/useIpcSubscriptions.ts | 35 ++++++++++++++++++- src/renderer/src/workspace/workspaceState.ts | 31 ++++++++++++++++ 3 files changed, 86 insertions(+), 1 deletion(-) diff --git a/src/renderer/src/workspace/hook/actions/initialHistory.ts b/src/renderer/src/workspace/hook/actions/initialHistory.ts index 9d207cca0..a4e41668a 100644 --- a/src/renderer/src/workspace/hook/actions/initialHistory.ts +++ b/src/renderer/src/workspace/hook/actions/initialHistory.ts @@ -188,6 +188,26 @@ export async function loadInitialHistoryForSession({ window.api.ghostAppend(sessionId, ghost) } + // Bootstrap-load equivalent of the live-ingest stamping in + // useIpcSubscriptions.ts. selectMergedEntries gates orphan + // ghost rendering against this timestamp; on resume we need + // it primed from the loaded JSONL tail so a ghost from the + // previous session whose updatedAt is older than the freshest + // loaded JSONL entry stays correctly hidden, while a ghost + // newer than every loaded entry (the + // "JSONL-stopped-mid-turn before the previous run died" case) + // surfaces as expected. + let lastJsonlEntryAt = current.lastJsonlEntryAt + for (const entry of initialEntries) { + const ts = (entry as { timestamp?: unknown }).timestamp + if (typeof ts !== 'string') continue + const ms = Date.parse(ts) + if (!Number.isFinite(ms)) continue + if (lastJsonlEntryAt === null || ms > lastJsonlEntryAt) { + lastJsonlEntryAt = ms + } + } + const nextRuntime = appendFeedDebugLog( { ...current, @@ -203,6 +223,7 @@ export async function loadInitialHistoryForSession({ toolUseIndex, toolResultIndex, ghosts: nextGhosts, + lastJsonlEntryAt, }, { layer: 'STATE', diff --git a/src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts b/src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts index 741313d56..a0609fb65 100644 --- a/src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts +++ b/src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts @@ -1108,6 +1108,36 @@ export function useIpcSubscriptions( ) : current.entries + // Track the newest JSONL entry timestamp this session has + // ever observed. selectMergedEntries uses this to decide + // whether an orphaned ghost is past the JSONL tail (render — + // proxy stalled past disk) or covered by it (hide — JSONL + // kept writing past this ghost, so it's a sidecar leak + // Claude Code never logs to its rollout). Comparison uses + // entry.timestamp (ISO 8601 on both Claude and Codex + // entries), NOT Date.now(), because on resume after a crash + // we want apples-to-apples wall-clock semantics with ghost + // _atp.updatedAt — both sides represent "when the producer + // observed this," so a yesterday-vs-yesterday comparison + // remains valid even if "now" is hours later. + // + // Non-conversation entries that lack `timestamp` (compact + // boundaries, queue ops) silently pass through the + // typeof-string guard without moving the cursor. That is + // the correct behaviour: those entries do not represent a + // fresh "JSONL is alive" signal at the wall clock the rest + // of the predicate cares about. + let lastJsonlEntryAt = current.lastJsonlEntryAt + for (const entry of appended) { + const ts = (entry as { timestamp?: unknown }).timestamp + if (typeof ts !== 'string') continue + const ms = Date.parse(ts) + if (!Number.isFinite(ms)) continue + if (lastJsonlEntryAt === null || ms > lastJsonlEntryAt) { + lastJsonlEntryAt = ms + } + } + // Ghost reconciliation — when authoritative entries land, // supersede any live ghost whose `(turnId, blockIndex)` // they replace. Runs per appended entry so ghost→real @@ -1149,6 +1179,7 @@ export function useIpcSubscriptions( // ghosts actually changed. Matches the treatment of // queuedMessages and the rest of this guard. const ghostsChanged = nextGhosts !== current.ghosts + const lastJsonlChanged = lastJsonlEntryAt !== current.lastJsonlEntryAt const noChange = appended.length === 0 && reconciledOptimisticText === null && @@ -1157,7 +1188,8 @@ export function useIpcSubscriptions( awaitingAssistant === current.awaitingAssistant && workContext === current.workContext && workActivity === current.workActivity && - !ghostsChanged + !ghostsChanged && + !lastJsonlChanged if (noChange) { closeSpan({ sessionId, @@ -1187,6 +1219,7 @@ export function useIpcSubscriptions( toolUseIndex, toolResultIndex, ghosts: nextGhosts, + lastJsonlEntryAt, }, { layer: 'JSONL', diff --git a/src/renderer/src/workspace/workspaceState.ts b/src/renderer/src/workspace/workspaceState.ts index 20efaff07..7b2e9edb6 100644 --- a/src/renderer/src/workspace/workspaceState.ts +++ b/src/renderer/src/workspace/workspaceState.ts @@ -376,6 +376,36 @@ export type SessionRuntime = { feedDebugLog: FeedDebugEntry[] feedDebugNextId: number feedDebugEpochMs: number | null + /** Wall-clock ms (epoch) of the newest JSONL entry timestamp we + * have observed for this session. + * + * WHY this exists separately from `entries[entries.length-1]`: + * Render decisions need a single comparable scalar against + * ghost `_atp.updatedAt`. Walking `entries` to find the max + * timestamp on every render would be O(N) per call; this is + * O(1) read after a single O(burst-size) update at ingest + * time. Equally important: the comparison must use entry + * *timestamp* (when the producer observed the event), not + * `Date.now()`. On resume after a crash the most recent + * JSONL entry might be from yesterday but ghost `updatedAt` + * is also from yesterday — both sides need to be in the + * same wall-clock universe or the comparison flips to + * nonsense. + * + * WHY null instead of 0 for "never seen": + * A 0-valued sentinel would make `ghost.updatedAt > 0` + * always true, accidentally rendering ghosts on a brand-new + * session that has produced nothing yet. Null is checked + * explicitly in `selectMergedEntries`. + * + * Used by `selectMergedEntries` (./mergedEntries.ts) to decide + * whether an orphaned ghost represents JSONL stalling past the + * proxy (render — proxy event is the only record) vs. a + * sidecar leak Claude Code never logs to its rollout (hide — + * JSONL kept writing real turns past it). See + * docs/superpowers/plans/2026-05-07-ghost-system-findings.md + * for the full rationale. */ + lastJsonlEntryAt: number | null /** Ghost-record state keyed by ghost uuid (`g--`). * * Ghosts are provisional ClaudeEntry records emitted from the @@ -476,6 +506,7 @@ export function emptyRuntime(): SessionRuntime { feedDebugLog: [], feedDebugNextId: 1, feedDebugEpochMs: null, + lastJsonlEntryAt: null, ghosts: new Map(), } } From f84f98ddc12f3aae50dff9bac8654486d0bb751d Mon Sep 17 00:00:00 2001 From: Julius Olsson Date: Thu, 7 May 2026 15:52:42 +0200 Subject: [PATCH 3/6] fix(feed): render orphan ghosts only when JSONL stalls past proxy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Layered predicate in selectMergedEntries: a ghost is render- eligible iff ALL of — 1. not superseded 2. orphaned (TTL elapsed without a JSONL match) 3. turnId != currentTurnId (SemanticStreamingTurn owns that) 4. _atp.updatedAt > lastJsonlEntryAt (proxy state past JSONL tail; structurally distinguishes "JSONL stalled past proxy" from "JSONL kept writing past this ghost") 5. not sidecar-shaped Why rules 4 AND 5, not just rule 4: the timestamp predicate alone fails for the tail-sidecar case (predict-next-prompt fires after the last real JSONL entry, no later real turn supersedes it via timestamp comparison). Rule 5 (single ≤200-char text block in an assistant turn) is the structural backstop. The trade-off — a real assistant turn that crashed before any JSONL write AND was a single short text block ("Done.") would also be hidden — is the same one commit 2a83978 made; this commit keeps the shape filter while adding the timestamp gate that 686b94e was missing. Bumps GHOST_ORPHAN_TTL_MS from 3000 to 30000. With the layered predicate, the orphan flag merely qualifies a ghost as render- eligible; rules 4 and 5 still gate visibility. So the TTL is now "how long before we conclude JSONL had its chance," not "how fast we paint fallback." 30s matches atp's library default and safely covers slow tool_results without prematurely qualifying mid-turn. Test cases land in scripts/test-ghost-fallback.ts (modified in-place, no new test files): 10 cases covering the predicate matrix — empty/superseded/unorphaned/current-turn skips, the timestamp gate (orphan older than JSONL tail = hide), the sidecar shape backstop (tail orphan with single ≤200-char text block = hide), substantive text rendering, tool_use rendering (not sidecar-shaped), fresh-session null-lastJsonlEntryAt path, and orphanStale TTL semantics. See docs/superpowers/plans/2026-05-07-ghost-system-findings.md for the long-form rationale and the case table this predicate satisfies. Co-Authored-By: Claude Opus 4.7 (1M context) --- scripts/test-ghost-fallback.ts | 192 +++++++++++-- .../workspace/hook/ipc/useIpcSubscriptions.ts | 15 +- src/renderer/src/workspace/mergedEntries.ts | 268 +++++++++--------- 3 files changed, 316 insertions(+), 159 deletions(-) diff --git a/scripts/test-ghost-fallback.ts b/scripts/test-ghost-fallback.ts index 790e40691..cd69826da 100644 --- a/scripts/test-ghost-fallback.ts +++ b/scripts/test-ghost-fallback.ts @@ -4,6 +4,7 @@ import { createGhost, orphanGhost, supersedeGhost, + type ClaudeContentBlock, type ClaudeEntry, } from 'agent-transcript-parser/ghost' @@ -11,13 +12,19 @@ import { orphanStale } from '../src/renderer/src/workspace/ghosts' import { selectMergedEntries } from '../src/renderer/src/workspace/mergedEntries' import { emptyRuntime } from '../src/renderer/src/workspace/workspaceState' -function baseEntry(uuid: string): ClaudeEntry { +// Wall-clock anchor for tests. Concrete values keep the predicate +// inputs explicit and avoid Date.now() drift between assertions. +const T_OLD_JSONL = 1_700_000_000_000 +const T_GHOST_NEWER = T_OLD_JSONL + 5_000 +const T_GHOST_OLDER = T_OLD_JSONL - 5_000 + +function baseEntry(uuid: string, timestamp = '2026-04-24T00:00:00.000Z'): ClaudeEntry { return { type: 'assistant', uuid, parentUuid: null, sessionId: 'session-1', - timestamp: '2026-04-24T00:00:00.000Z', + timestamp, message: { id: `msg-${uuid}`, role: 'assistant', @@ -26,16 +33,55 @@ function baseEntry(uuid: string): ClaudeEntry { } as ClaudeEntry } -function textGhost(turnId: string, text: string) { +function ghostWithContent( + turnId: string, + content: ClaudeContentBlock[], + now: number, +) { return createGhost({ sessionId: 'session-1', turnId, blockIndex: 0, role: 'assistant', - content: [{ type: 'text', text }], + content, + now, }) } +function shortTextGhost(turnId: string, text: string, now: number) { + return ghostWithContent(turnId, [{ type: 'text', text }], now) +} + +function longTextGhost(turnId: string, now: number) { + // 250 chars — past the SIDECAR_GHOST_TEXT_MAX threshold (200), so + // the sidecar shape filter will let it through. + const text = 'A real assistant turn that streamed enough text that it could not be a one-line title-gen sidecar leak. The shape filter is a 200-char cap; this string deliberately blows past that to verify the structural rule applies.' + return ghostWithContent(turnId, [{ type: 'text', text }], now) +} + +function toolUseGhost(turnId: string, now: number) { + // Tool_use ghosts are explicitly NOT sidecar-shaped (the shape + // filter only matches single-text-block assistant ghosts), so a + // tool_use orphan past lastJsonlEntryAt should render even if + // short. + return ghostWithContent( + turnId, + [ + { + type: 'tool_use', + id: `tool-${turnId}`, + name: 'Read', + input: { file_path: '/tmp/example.ts' }, + }, + ], + now, + ) +} + +// --------------------------------------------------------------------------- +// 1. Empty ghost map → return entries by identity (reference-stable +// short-circuit; Feed memos rely on this). +// --------------------------------------------------------------------------- { const runtime = emptyRuntime() runtime.entries = [baseEntry('real-1')] @@ -43,52 +89,164 @@ function textGhost(turnId: string, text: string) { assert.equal(merged, runtime.entries) } +// --------------------------------------------------------------------------- +// 2. Ghost not yet orphaned → hidden (JSONL might still arrive +// within TTL; SemanticStreamingTurn covers the live current +// turn, this ghost shouldn't double-render). +// --------------------------------------------------------------------------- { const runtime = emptyRuntime() runtime.entries = [baseEntry('real-1')] - const ghost = textGhost('turn-a', 'not ready yet') + runtime.lastJsonlEntryAt = T_OLD_JSONL + const ghost = longTextGhost('turn-a', T_GHOST_NEWER) runtime.ghosts = new Map([[ghost.uuid, ghost]]) + const merged = selectMergedEntries(runtime, null) + assert.equal(merged, runtime.entries, 'unorphaned ghost must not render') +} +// --------------------------------------------------------------------------- +// 3. Superseded ghost → hidden (JSONL caught up; reconcileUpstream +// has the row already). +// --------------------------------------------------------------------------- +{ + const runtime = emptyRuntime() + runtime.entries = [baseEntry('real-1')] + runtime.lastJsonlEntryAt = T_OLD_JSONL + const ghost = orphanGhost( + supersedeGhost(longTextGhost('turn-a', T_GHOST_NEWER), 'real-1', T_GHOST_NEWER + 1), + T_GHOST_NEWER + 2, + ) + runtime.ghosts = new Map([[ghost.uuid, ghost]]) const merged = selectMergedEntries(runtime, null) - assert.equal(merged, runtime.entries) + assert.equal(merged, runtime.entries, 'superseded ghost must not render') } +// --------------------------------------------------------------------------- +// 4. Orphan ghost for the live current turn → hidden. +// SemanticStreamingTurn owns the live turn render; surfacing a +// ghost for the same turn would double-render. +// --------------------------------------------------------------------------- { const runtime = emptyRuntime() runtime.entries = [baseEntry('real-1')] - const ghost = orphanGhost(textGhost('turn-a', 'jsonl never wrote'), 2000) + runtime.lastJsonlEntryAt = T_OLD_JSONL + const ghost = orphanGhost( + longTextGhost('turn-a', T_GHOST_NEWER), + T_GHOST_NEWER + 1, + ) runtime.ghosts = new Map([[ghost.uuid, ghost]]) + const merged = selectMergedEntries(runtime, 'turn-a') + assert.equal(merged, runtime.entries, 'current-turn ghost must not render') +} +// --------------------------------------------------------------------------- +// 5. Orphan ghost OLDER than lastJsonlEntryAt → hidden. This is the +// "stale orphan from earlier in the session, JSONL kept writing +// past it" case: a sidecar leak that happened before later real +// JSONL entries. Predicate-4 catches it structurally. +// --------------------------------------------------------------------------- +{ + const runtime = emptyRuntime() + runtime.entries = [baseEntry('real-1')] + runtime.lastJsonlEntryAt = T_OLD_JSONL + const ghost = orphanGhost( + longTextGhost('turn-a', T_GHOST_OLDER), + T_GHOST_OLDER + 1, + ) + runtime.ghosts = new Map([[ghost.uuid, ghost]]) const merged = selectMergedEntries(runtime, null) - assert.equal(merged.length, 2) - assert.equal(merged[1]?.uuid, ghost.uuid) + assert.equal(merged, runtime.entries, 'orphan older than JSONL tail must not render') } +// --------------------------------------------------------------------------- +// 6. Orphan ghost NEWER than lastJsonlEntryAt with sidecar shape +// (single short text block, ≤200 chars) → hidden. This is the +// tail-sidecar case the timestamp predicate alone cannot +// catch: predict-next-prompt fires after the last real JSONL +// entry, no later real turn supersedes it. Shape filter is the +// backstop. +// --------------------------------------------------------------------------- { const runtime = emptyRuntime() runtime.entries = [baseEntry('real-1')] - const ghost = orphanGhost(textGhost('turn-a', 'current live turn'), 2000) + runtime.lastJsonlEntryAt = T_OLD_JSONL + const ghost = orphanGhost( + shortTextGhost('turn-a', 'short ack', T_GHOST_NEWER), + T_GHOST_NEWER + 1, + ) runtime.ghosts = new Map([[ghost.uuid, ghost]]) + const merged = selectMergedEntries(runtime, null) + assert.equal(merged, runtime.entries, 'sidecar-shaped tail orphan must not render') +} - const merged = selectMergedEntries(runtime, 'turn-a') - assert.equal(merged, runtime.entries) +// --------------------------------------------------------------------------- +// 7. Orphan ghost NEWER than lastJsonlEntryAt with substantive text +// (>200 chars) → renders. This is the main case ghost was built +// for: JSONL stopped writing while proxy kept going for an +// in-flight assistant turn. +// --------------------------------------------------------------------------- +{ + const runtime = emptyRuntime() + runtime.entries = [baseEntry('real-1')] + runtime.lastJsonlEntryAt = T_OLD_JSONL + const ghost = orphanGhost( + longTextGhost('turn-a', T_GHOST_NEWER), + T_GHOST_NEWER + 1, + ) + runtime.ghosts = new Map([[ghost.uuid, ghost]]) + const merged = selectMergedEntries(runtime, null) + assert.equal(merged.length, 2, 'substantive orphan past JSONL tail must render') + assert.equal(merged[1]?.uuid, ghost.uuid) } +// --------------------------------------------------------------------------- +// 8. Orphan tool_use ghost NEWER than lastJsonlEntryAt → renders +// even though short. tool_use is structurally not a sidecar +// shape — Claude Code's auxiliary calls all return text-only +// bodies, never tool_use blocks. +// --------------------------------------------------------------------------- { const runtime = emptyRuntime() runtime.entries = [baseEntry('real-1')] + runtime.lastJsonlEntryAt = T_OLD_JSONL const ghost = orphanGhost( - supersedeGhost(textGhost('turn-a', 'already committed'), 'real-1', 2000), - 3000, + toolUseGhost('turn-a', T_GHOST_NEWER), + T_GHOST_NEWER + 1, ) runtime.ghosts = new Map([[ghost.uuid, ghost]]) + const merged = selectMergedEntries(runtime, null) + assert.equal(merged.length, 2, 'tool_use orphan past JSONL tail must render') + assert.equal(merged[1]?.uuid, ghost.uuid) +} +// --------------------------------------------------------------------------- +// 9. lastJsonlEntryAt === null (fresh session, no JSONL ever) AND +// orphan ghost present → render decision falls through the +// timestamp gate (rule 4 only fires when lastJsonlEntryAt is +// non-null). The shape filter still applies, so the ghost +// renders only if it's not sidecar-shaped. +// --------------------------------------------------------------------------- +{ + const runtime = emptyRuntime() + runtime.entries = [baseEntry('real-1')] + runtime.lastJsonlEntryAt = null + const ghost = orphanGhost( + longTextGhost('turn-a', T_GHOST_NEWER), + T_GHOST_NEWER + 1, + ) + runtime.ghosts = new Map([[ghost.uuid, ghost]]) const merged = selectMergedEntries(runtime, null) - assert.equal(merged, runtime.entries) + assert.equal(merged.length, 2, 'orphan with null lastJsonlEntryAt must render if shape passes') + assert.equal(merged[1]?.uuid, ghost.uuid) } +// --------------------------------------------------------------------------- +// 10. orphanStale TTL behavior. Confirms the helper still produces +// reference-stable no-ops and that an orphan flag is set when +// the threshold elapses. +// --------------------------------------------------------------------------- { - const fresh = textGhost('turn-a', 'fresh') + const fresh = shortTextGhost('turn-a', 'fresh', T_OLD_JSONL) const freshMap = new Map([[fresh.uuid, fresh]]) assert.equal(orphanStale(freshMap, fresh._atp.updatedAt + 500, 1000), freshMap) @@ -97,4 +255,4 @@ function textGhost(turnId: string, text: string) { assert.ok(staleMap.get(fresh.uuid)?._atp.orphanedAt !== undefined) } -console.log('ghost fallback tests passed') +console.log('ghost render predicate tests passed') diff --git a/src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts b/src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts index a0609fb65..45c03bcc5 100644 --- a/src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts +++ b/src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts @@ -94,7 +94,20 @@ type WorktreeCacheEntry = { const WORKTREE_CACHE_TTL_MS = 5000 const WORK_CONTEXT_RECENT_RAW_LIMIT = 500 -const GHOST_ORPHAN_TTL_MS = 3000 +// Threshold before an unsuperseded ghost is marked orphaned. +// +// 3000ms was the value used while orphan rendering was always-on +// (commit 686b94e) — the goal was to surface fallback fast. With +// the layered predicate in mergedEntries.ts an orphan flag merely +// makes a ghost ELIGIBLE for rendering; rules 4 (timestamp gate) +// and 5 (sidecar shape) still gate final visibility. So the TTL +// is now "how long do we wait before concluding JSONL had its +// chance for this ghost," not "how fast do we paint." 30000ms +// matches atp's library default and safely covers slow tool +// results (large Read, large Bash output, slow fetches) without +// prematurely qualifying a ghost mid-turn. Sweep cadence stays at +// 1s — that's the polling rate, not the threshold. +const GHOST_ORPHAN_TTL_MS = 30000 const GHOST_ORPHAN_SWEEP_MS = 1000 function canIngestWorkContext(raw: unknown, hasWorktreeCache: boolean): boolean { diff --git a/src/renderer/src/workspace/mergedEntries.ts b/src/renderer/src/workspace/mergedEntries.ts index 7a3e6d1f2..900bd887f 100644 --- a/src/renderer/src/workspace/mergedEntries.ts +++ b/src/renderer/src/workspace/mergedEntries.ts @@ -1,186 +1,172 @@ -// Render-time selector: merge upstream entries with current ghost -// state into the single Entry list Feed consumes. +// Render-time selector: decide which (if any) ghost entries get +// merged into the rendered feed. // -// WHY this lives in its own file: -// The merge is called from TileLeaf's render path, and memoization -// needs stable references. Putting the selector alongside the -// ghost reducer keeps the integration surface tiny — TileLeaf -// imports one function from one place; workspaceStore imports -// reducer helpers from `./ghosts`; Feed imports nothing about -// ghosts at all. That separation is load-bearing: when Phase 3 -// deletes the old "two owners" branch in Feed.tsx, Feed will -// simply consume the merged list with no awareness of ghosts. +// ----------------------------------------------------------------------------- +// Where ghost rendering fits in cc-shell today +// ----------------------------------------------------------------------------- // -// The merge is a thin wrapper around atp's `mergeWithUpstream` so -// the provenance and semantics stay documented in one place (the -// atp doc). This file adds nothing to the merge rules; it only -// chooses atp's default options and threads the runtime's state. - -import type { Entry } from '@shared/types/transcript' -import type { SessionRuntime } from '@renderer/workspace/workspaceState' -import type { ClaudeContentBlock, GhostEntry } from 'agent-transcript-parser/ghost' -import { mergeWithUpstream } from 'agent-transcript-parser/ghost' - -// Sidecar-shape suppression — the symptom side of a known proxy-filter gap. +// The live current turn is rendered by `SemanticStreamingTurn` +// directly off `runtime.semantic.currentTurn`, NOT through ghosts. +// JSONL writes catch up within ~100 ms via `reconcileUpstream` and +// supersede the ghost. Most ticks, the ghost map has no rendered +// output here at all. +// +// The one case ghost rendering exists for is JSONL stalling past +// the proxy. Two situations produce that: +// 1. Live-stuck: the agent process gets wedged or its writer +// backlogs while the proxy keeps emitting events. Eventually +// the orphan TTL fires and the ghost is the only record of +// what happened for that turn. +// 2. Resume-after-crash: ghost log on disk has events past the +// newest JSONL entry; JSONL never caught up before the +// previous run died. On bootstrap, that ghost surfaces as the +// lost partial turn. // -// WHAT this is fighting: -// Claude Code routes auxiliary API calls (title gen, branch-name gen, -// "predict the user's next prompt" follow-ups) through the same -// /v1/messages endpoint as real turns. ClaudeProxyAdapter sees them as -// live flows, publishes turn_started → text_delta → turn_completed -// semantic events, ghostsFromSemanticTurn mints a ghost block. Claude -// Code does NOT write these calls to its JSONL rollout, so -// reconcileUpstream never matches a real entry, the orphan timeout -// fires (~30s later), and mergeWithUpstream appends the ghost to the -// tail of the feed. Each title-gen call leaves a 2-10 word fragment -// permanently parked at the bottom of the conversation. +// ----------------------------------------------------------------------------- +// The layered predicate +// ----------------------------------------------------------------------------- // -// WHY this lives in cc-shell instead of the headless filter: -// ClaudeProxyAdapter already has a sidecar predicate (see commit -// c8c2623, May 2026) that demotes flows by request body shape: -// - max_tokens ≤ 1024 AND messageCount ≤ 3, OR -// - one of four known system-prompt prefixes -// That predicate misses the "predict-next-prompt" feature because the -// call carries the full conversation history (messageCount > 3) and -// uses a system prompt not in the prefix list. Forensic confirmation -// from debug bundle 2026-05-07T08-26-35-212-5d948ab5: 7 orphan ghosts -// visible at the bottom of the feed, none of 23 flows demoted, all -// sidecar turns 12-41 chars and 188-602ms — well below any real -// assistant turn (≥76 chars, ≥808ms in the same bundle). +// A ghost is render-eligible iff ALL of: +// 1. Not superseded. +// 2. Orphaned (TTL has elapsed without a JSONL match). +// 3. `turnId !== currentTurnId` (SemanticStreamingTurn owns that). +// 4. `_atp.updatedAt > lastJsonlEntryAt` (proxy state is past the +// JSONL tail; structurally distinguishes the live-stuck case +// from "ghost from earlier in the session that JSONL kept +// writing past"). +// 5. Not sidecar-shaped. // -// Fixing the predicate properly requires the actual request body to -// know the new system prompt template, which we don't have here. -// This filter eliminates the symptom by detecting the SHAPE of the -// leaked turn (short, single-block, plain text, assistant role, -// orphaned) at render time. +// Why rules 4 AND 5, not just rule 4: +// The timestamp predicate is structurally correct for "stale +// orphan from earlier in the session below newer commits," but +// it cannot tell apart these two TAIL cases: +// a) JSONL stopped mid-turn, ghost has the lost partial turn +// (should render). +// b) Last real JSONL entry committed at t=100, then a sidecar +// leak (predict-next-prompt / title-gen / branch-name) at +// t=105 with no later real turn to supersede it (should +// NOT render). +// Both have ghost.updatedAt > lastJsonlEntryAt and produce a +// tail orphan with no JSONL counterpart. Rule 5 is a structural +// shape check that matches Claude Code's known sidecar +// fingerprint: assistant role + single text block + ≤200 chars. +// Predict-next-prompt with full conversation history can exceed +// the proxy-side budget predicate, but the response body is +// short and shaped this way; matching at render time is the +// backstop. // -// Trade-off / false-positive risk: -// A genuinely brief assistant turn (e.g. "Done.") whose JSONL never -// committed would also be suppressed. Acceptable because: -// a) Orphans are already an error path — JSONL not landing means -// upstream Claude Code crashed mid-turn. -// b) The 200-char threshold is wider than any observed sidecar in -// production bundles, so it errs toward keeping real content -// visible at the cost of letting through marginal sidecars. -// c) Permanent clutter from 6+ orphan titles is a daily UX harm; -// a hidden 4-char crash response is a once-in-a-rare-while -// invisible loss. +// Trade-off (knowingly accepted): a real assistant turn that +// crashed before any JSONL write AND was a single short text +// block (e.g. "Done.") would also be hidden. In production, that +// loss is rare; the alternative (orphan title-gen / next-prompt +// fragments piling up at the bottom of every session) is a daily +// UX harm. The trade is the same one commit 2a83978 made; this +// file keeps the shape filter while adding the timestamp gate +// that 686b94e was missing. // -// Future work: -// When we capture a live request body for the predict-next-prompt -// feature, extend SIDECAR_SYSTEM_PROMPT_PREFIXES in claude-code-headless -// and the body-shape predicate. At that point this renderer-side -// filter becomes redundant and can be deleted. +// ----------------------------------------------------------------------------- +// Reference stability (load-bearing for Feed memos) +// ----------------------------------------------------------------------------- +// +// Feed's row memos key off the `entries` array IDENTITY. When no +// ghost survives the predicate, this selector returns +// `runtime.entries` by identity, NOT a fresh `[...entries]`. The +// pre-fix `mergeWithUpstream` always returned `[...upstream, +// ...trailing]` even when `trailing` was empty, busting memos on +// every tick. atp's `mergeWithUpstream` is only called when +// `visible.size > 0`. +// +// Future work, not done here: +// Phase 3 of the original headless redesign (delete +// SemanticStreamingTurn, render the live current turn through +// ghosts + ordered insertion in `mergeWithUpstream`) requires +// atp learning to anchor by parentUuid / turnId / nearest +// committed neighbor instead of always tail-appending. Until +// that ships, the live current turn stays owned by +// SemanticStreamingTurn and ghost rendering is reserved for the +// JSONL-stalled-past-proxy fallback case described above. + +import type { Entry } from '@shared/types/transcript' +import type { SessionRuntime } from '@renderer/workspace/workspaceState' +import type { ClaudeContentBlock, GhostEntry } from 'agent-transcript-parser/ghost' +import { mergeWithUpstream } from 'agent-transcript-parser/ghost' -// Cap chosen empirically against this user's debug bundle (max sidecar -// turn = 41 chars; min real assistant turn = 76 chars). 200 leaves -// headroom for slightly longer titles and a generous safety margin -// before we'd start cutting into real prose. +// Cap chosen empirically against debug bundle 2026-05-07T08-26-35 +// (max sidecar turn = 41 chars, min real assistant turn = 76 +// chars). 200 leaves headroom for slightly-longer next-prompt +// variants and a generous safety margin before we'd start cutting +// into real prose. Same constant lived in mergedEntries.ts at +// commit 2a83978; reused here with the same threshold so the +// regression surface is unchanged. const SIDECAR_GHOST_TEXT_MAX = 200 function ghostHasSidecarShape(ghost: GhostEntry): boolean { - // Only assistant orphans match — sidecars come back through the proxy - // as assistant streams. User entries don't ghost (they get optimistic - // user rows reconciled separately) and don't have this failure mode. + // Only assistant orphans match. Sidecars come back through the + // proxy as assistant streams; user entries don't ghost (they get + // optimistic-row reconciliation separately) and don't have this + // failure mode. if (ghost.message?.role !== 'assistant') return false const content = ghost.message?.content if (!Array.isArray(content)) return false - // Single block. Real turns from a healthy proxy stream nearly always - // carry at least a tool_use companion or fragmentation across blocks - // even when short; a lone text block is the title-gen fingerprint. + // Single block. Real assistant turns from a healthy proxy stream + // nearly always carry at least a tool_use companion or + // fragmentation across blocks even when short; a lone text block + // is the title-gen / predict-next-prompt fingerprint. if (content.length !== 1) return false const block = content[0] as ClaudeContentBlock if (block.type !== 'text') return false - // `text` is required on text blocks per atp's content type, but be - // defensive — a malformed ghost should not crash the feed selector. + // `text` is required on text blocks per atp's content type, but + // be defensive — a malformed ghost should not crash the feed + // selector. const text = (block as { text?: unknown }).text if (typeof text !== 'string') return false return text.length <= SIDECAR_GHOST_TEXT_MAX } /** - * Merged entry list for rendering. Superseded ghosts are dropped; - * orphaned ghosts stay visible (they're the only record of the - * block). - * - * Reference stability: - * - if `runtime.ghosts` is empty OR every ghost in the map has - * been superseded, returns `runtime.entries` as-is. That short- - * circuit is what keeps Feed's useMemo([entries]) chain from - * thrashing once the live turn is done and all ghosts have - * handed off to the committed JSONL entries. - * - otherwise returns a fresh array from mergeWithUpstream. Feed's - * memos recompute, which is correct: there is at least one - * provisional ghost visible and the list is genuinely different - * from `runtime.entries`. - * - * WHY not always call mergeWithUpstream: the previous implementation - * did, and even when `trailing` ended up empty, the return was still - * `[...upstream, ...[]]` — a fresh array reference. Downstream memos - * keyed on entries identity recomputed on every tick, so the - * "memoization path is unchanged" promise in the old docstring was - * only theoretical. + * Render-time merge of `runtime.entries` with the surviving ghost + * set. See the file-level WHY block above for the predicate + * design. */ export function selectMergedEntries( runtime: SessionRuntime, currentTurnId: string | null, ): Entry[] { - const { ghosts, entries } = runtime + const { ghosts, entries, lastJsonlEntryAt } = runtime if (ghosts.size === 0) return entries - // Render ghosts only after the orphan timeout has fired. Plain - // unsuperseded ghosts are the normal handoff window between live - // semantic streaming and JSONL; showing them immediately was the - // source of duplicate/wrong-order feed rows. Orphaned ghosts are - // different: the authoritative JSONL has not arrived in time, so - // the ghost is the only surviving record of what the proxy observed. - // - // Current-turn ghosts are still excluded because - // SemanticStreamingTurn owns the active live view. When that turn - // completes and currentTurn becomes null, orphaned ghosts can step - // in as the fallback transcript. - const visibleGhosts = new Map() + const visible = new Map() for (const [uuid, ghost] of ghosts) { if (ghost._atp.supersededBy !== undefined) continue if (ghost._atp.orphanedAt === undefined) continue - if (currentTurnId !== null && ghost._atp.turnId === currentTurnId) { - continue - } - // Suppress orphan ghosts that match Claude Code's sidecar - // (title-gen / predict-next-prompt) shape. See the - // ghostHasSidecarShape doc-comment above for the full rationale — - // short version: these calls leak through the proxy, never land in - // JSONL, get orphaned, and pile up at the bottom of the feed - // forever. Detecting them by content shape is a renderer-side - // workaround until the headless body-shape predicate is widened. + if (currentTurnId !== null && ghost._atp.turnId === currentTurnId) continue + // Rule 4: only ghosts past the JSONL tail. Null + // lastJsonlEntryAt (fresh session, never observed any JSONL + // entry) falls through — rule 5 still applies, and the rest + // of the predicate (orphaned + non-current) keeps this + // narrow. + if (lastJsonlEntryAt !== null && ghost._atp.updatedAt <= lastJsonlEntryAt) continue if (ghostHasSidecarShape(ghost)) continue - visibleGhosts.set(uuid, ghost) + visible.set(uuid, ghost) } - if (visibleGhosts.size === 0) return entries + if (visible.size === 0) return entries - return mergeWithUpstream(entries, visibleGhosts, { + // Tail-append is correct for the surviving set: every visible + // ghost is by predicate-3 not the active turn and by predicate-4 + // newer than every committed entry, so chronologically it + // belongs at the very end. + return mergeWithUpstream(entries, visible, { trustSupersededFlag: true, }) as Entry[] } /** * Whether Feed should render the live `SemanticStreamingTurn` - * component. True iff there is a current turn — the merged feed - * selector (`selectMergedEntries`) suppresses ghosts from the main - * feed, so SemanticStreamingTurn has exclusive ownership of the - * live view and there is no duplicate-render risk. - * - * WHY the old predicate was wrong: - * It returned false the moment any un-superseded ghost for the - * current turn existed — which happens on every block transition, - * because tool_use ghosts are minted BEFORE their upstream entry - * commits. The live view flicker/null-flip at id:131 in the - * 2026-04-20 rendering-fixes evidence log was that gate tripping. - * - * Phase 3 of the original headless redesign will delete - * SemanticStreamingTurn entirely and render everything through - * ghosts + the merged feed. Until then, single-ownership in the - * main feed is the surgical fix. + * component. True iff there is a current turn — full stop. The + * merged feed selector (`selectMergedEntries`) hides ghosts whose + * `turnId === currentTurnId`, so SemanticStreamingTurn has + * exclusive ownership of the live view and there is no + * duplicate-render risk. */ export function shouldShowSemanticStreaming(runtime: SessionRuntime): boolean { return runtime.semantic.currentTurn !== null From 9777670a788c0784890816f436593b0293435cac Mon Sep 17 00:00:00 2001 From: Julius Olsson Date: Thu, 7 May 2026 16:00:44 +0200 Subject: [PATCH 4/6] docs(ghost): align inline comments with current behavior Strips the "transcript-first feed with a ghost overlay" / "Phase 3 will delete SemanticStreamingTurn" framings that described an unfinished plan. Replaces with the actual current behavior: ghost is a bookkeeping ledger that surfaces in the rendered feed only when the layered predicate in selectMergedEntries fires (JSONL stalled past proxy + not sidecar-shaped). The live current turn is owned by SemanticStreamingTurn directly off the semantic channel. ghosts.ts file-level block: rewrites "WHY this exists" to describe the current two-consumer model (selectMergedEntries fallback + ghostJournal persistence) instead of the unfinished single-owner overlay. Adds reference to the findings doc. TileLeaf.tsx call-site comments: updates the entries={} and semanticTurn={} prop descriptions to match the layered predicate. No code change. The full rationale is in docs/superpowers/plans/2026-05-07-ghost-system-findings.md. Co-Authored-By: Claude Opus 4.7 (1M context) --- src/renderer/src/workspace/ghosts.ts | 100 ++++++++++-------- .../src/workspace/tile-tree/TileLeaf.tsx | 34 +++--- 2 files changed, 71 insertions(+), 63 deletions(-) diff --git a/src/renderer/src/workspace/ghosts.ts b/src/renderer/src/workspace/ghosts.ts index 90c4fe440..982a6b9be 100644 --- a/src/renderer/src/workspace/ghosts.ts +++ b/src/renderer/src/workspace/ghosts.ts @@ -1,62 +1,72 @@ -// Ghost-record bridge between the live semantic reducer and the -// durable JSONL transcript. +// Renderer-side ghost reducer + the bridge between the live +// semantic stream, the durable JSONL transcript, and disk +// persistence. // // ----------------------------------------------------------------------------- -// WHY this exists +// What ghost is in cc-shell today // ----------------------------------------------------------------------------- // -// The Feed used to have two independent owners of visible assistant text: +// Ghost is a parallel disk-backed ledger of semantic events. As +// the proxy stream emits events, this file mints provisional +// `ClaudeEntry` records via atp's `createGhost`. When the +// authoritative JSONL entry lands (Claude's batched 100 ms drain; +// Codex's mpsc flush), `reconcileUpstream` matches by message.id / +// codexTurnId / tool_use_id and supersedes the ghost. If JSONL +// never matches (Claude Code's auxiliary calls — title gen, +// predict-next-prompt, branch-name gen — are not written to the +// rollout), `orphanStale` flags the ghost after the TTL. // -// 1. committed transcript entries read from disk JSONL (`runtime.entries`) -// 2. the live semantic tail (`runtime.semantic.currentTurn`) +// The live current turn renders via `SemanticStreamingTurn` +// directly off `runtime.semantic.currentTurn` — NOT through +// ghosts. So most ticks, the ghost map has no rendered output: +// the work this file does is bookkeeping (mint, reconcile, +// orphan, gc, persist) for two consumers: // -// Feed rendered both unconditionally — upstream entries first, then the -// live tail below. That worked while upstream's write was same-tick with -// the live event, which it is NOT. Upstream Claude Code writes transcript -// JSONL fire-and-forget on a 100 ms batched drain (10 ms for remote -// sessions), and Codex's RolloutRecorder queues writes through a tokio -// mpsc to a background task that persists on explicit flush barriers. -// Both CLIs intentionally let the live view lead, then eventually catch -// up. +// 1. `selectMergedEntries` (./mergedEntries.ts) — surfaces +// orphan ghosts ONLY when JSONL has stalled past the proxy +// (live-stuck mid-turn or resume-after-crash with partial +// JSONL). The layered predicate there has the full design +// rationale. // -// That design is fine for the CLI's own TUI, which is the ONLY consumer -// and knows never to show both views at once. It breaks for cc-shell, -// which reads the durable JSONL AND consumes the live semantic channel -// and must merge the two into one feed. The observed symptom was a -// duplicate-render bug on Codex: committed assistant entry plus live -// rollout semantic turn rendered the same sentence twice. +// 2. `ghostJournal.ts` (main process) — append-only JSONL log +// under /ghost-logs/.ghost.jsonl. +// Survives reload / restart so the JSONL-stuck case can +// recover the lost partial turn on resume via the bootstrap +// merge in src/renderer/src/workspace/hook/actions/session.ts. // -// The fix is a transcript-first feed with a ghost overlay. Ghosts are -// minted from live semantic state as a provisional `ClaudeEntry` with -// `_atp.origin = 'ghost'`, reconciled against upstream entries as they -// land, and merged at read time via `mergeWithUpstream`. Once the real -// entry exists, the ghost is superseded and drops out of the merged -// list. If the real entry never arrives, the ghost is orphaned and -// stays visible as the sole record of what the live layer saw. +// ----------------------------------------------------------------------------- +// Provider-aware reconciliation +// ----------------------------------------------------------------------------- // -// See `agent-transcript-parser/docs/ghost.md` for the full primitive, -// and the 2026-04-18 plan in cc-shell's docs for the integration. +// Claude: upstream assistant entries carry `message.id == turnId`, +// so one upstream entry supersedes every ghost block for that +// turn at once. +// +// Codex: rollout emits one entry per content block, with the +// rollout response_id stamped onto the mapped entry by +// `stampCodexTurnId` in ../codex/rollout.ts. Match is by +// (turnId, blockIndex). When that fails, both providers fall back +// to tool_use_id / call_id pairing for tool blocks. // // ----------------------------------------------------------------------------- -// Layering +// Reference stability // ----------------------------------------------------------------------------- // -// atp — agent-agnostic primitives (`createGhost`, -// `updateGhost`, `supersedeGhost`, `orphanGhost`, -// `reduceGhostLog`, `mergeWithUpstream`) -// cc-shell/ghosts.ts — THIS FILE. Pure functions that bridge -// `SemanticLiveTurn` → ghost entries and fold -// upstream ingest into supersedes. -// workspaceStore.ts — calls the two functions here at the semantic -// fold site and the JSONL ingest site. Disk -// persistence (Phase 2) is owned by -// `src/main/ghostJournal.ts`. -// Feed.tsx — consumes the merged Entry list via a derived -// selector; never sees ghost/non-ghost -// distinction at the component level. +// Every reducer below MUST return `prev` unchanged on no-op so +// React memoization holds at the call site. This isn't an +// optimization — it's load-bearing. The pre-fix versions always +// allocated `new Map(prev)` at the top, which made +// `nextGhosts !== current.ghosts` always true downstream, forcing +// a setRuntimes cascade that busted every useMemo([entries]) in +// Feed via selectMergedEntries. +// +// See `agent-transcript-parser/docs/ghost.md` for the underlying +// primitive's semantics, and +// docs/superpowers/plans/2026-05-07-ghost-system-findings.md for +// the longer story of how cc-shell got here. // -// No function here performs IO. No function here subscribes to events. -// Input goes in, new `Map` comes out. +// No function here performs IO. No function here subscribes to +// events. Input goes in, new `Map` comes out. import { createGhost, diff --git a/src/renderer/src/workspace/tile-tree/TileLeaf.tsx b/src/renderer/src/workspace/tile-tree/TileLeaf.tsx index 496eb9fe6..47162c725 100644 --- a/src/renderer/src/workspace/tile-tree/TileLeaf.tsx +++ b/src/renderer/src/workspace/tile-tree/TileLeaf.tsx @@ -333,14 +333,16 @@ export function TileLeaf({ sessionId={sessionId} provider={provider} workspaceRoot={workspace.state.sessions[sessionId]?.cwd ?? null} - // Merged upstream + ghost feed. Split ownership: the live - // turn's ghosts are EXCLUDED from the merge (the live view - // component below is the sole owner of the current turn); - // earlier turns' orphaned or still-unreconciled ghosts fall - // through. The `currentTurnId` argument is what encodes that - // split — see `./mergedEntries.ts` for the filter and - // docs/superpowers/plans/2026-04-20-rendering-fixes.md - // Task 5 for the WHY. + // Committed transcript + (rare) orphan-ghost fallback. + // The layered predicate in selectMergedEntries renders + // a ghost only when JSONL has stalled past the proxy + // AND the ghost is not sidecar-shaped (title-gen / + // predict-next-prompt fingerprint). The live current + // turn is owned by `SemanticStreamingTurn` below; the + // `currentTurnId` argument hides any ghost for that + // turn so the two surfaces never double-render. + // See ./mergedEntries.ts and + // docs/superpowers/plans/2026-05-07-ghost-system-findings.md. entries={selectMergedEntries( runtime, runtime.semantic.currentTurn?.turnId ?? null, @@ -364,16 +366,12 @@ export function TileLeaf({ streamPhasePendingToolUseId={runtime.streamPhasePendingToolUseId} turnStartedAt={runtime.turnStartedAt} // Live-turn ownership: SemanticStreamingTurn renders the - // current turn end-to-end. Earlier un-reconciled ghosts - // are now filtered OUT of the merged feed (see the - // `currentTurnId` argument to selectMergedEntries above), - // so there is no double-render risk — the two surfaces - // own different time slices. `shouldShowSemanticStreaming` - // collapses to "is there a current turn?" as a result. - // See - // docs/superpowers/plans/2026-04-20-rendering-fixes.md - // Task 5 and - // docs/superpowers/plans/2026-04-17-claude-semantic-provider-gating.md. + // current turn end-to-end off the semantic channel. + // Ghosts for the same turnId are filtered out of the + // merged feed by selectMergedEntries (currentTurnId + // argument), so there is no double-render risk. + // shouldShowSemanticStreaming collapses to "is there a + // current turn?". semanticTurn={ shouldShowSemanticStreaming(runtime) ? runtime.semantic.currentTurn From 5726ed8d1144a797024fc8abdedbf7bbf8a17f6a Mon Sep 17 00:00:00 2001 From: Julius Olsson Date: Fri, 8 May 2026 09:55:35 +0200 Subject: [PATCH 5/6] docs(design): add docs/design/ as source-of-truth location, with ghost-system.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Establishes docs/design/ as the canonical home for evergreen design docs that explain non-obvious subsystems. The README in that directory states the convention: these files are sources of truth, NOT changelogs or running notes; do not change them lightly. Date-stamped plans and forensic write-ups continue to live in docs/superpowers/plans/. docs/design/ghost-system.md is the first such doc. It explains: * what ghost is in plain English (the bridge between the proxy semantic stream and the durable JSONL transcript) * why ghost exists (none of the naive alternatives — JSONL-only, proxy-only, both unconditionally — work) * the dual-owner relationship with SemanticStreamingTurn that makes rule 3 of the predicate load-bearing * the architecture (ASCII diagram + per-file role table) * the lifecycle (created → updated → superseded / orphaned) * the five-rule render predicate, reproduced canonically * provider-aware reconciliation matchers * reference-stability invariants * persistence + resume behavior * the orphan TTL semantics * Phase 3 explicitly out of scope * a Warning section listing the four prior fix attempts that regressed and the specific things future authors must NOT drop without re-thinking the whole subsystem Inline comments in every ghost-touching file now point at the design doc as the canonical explanation: * src/renderer/src/workspace/ghosts.ts (file-level block) * src/renderer/src/workspace/mergedEntries.ts (file-level block) * src/renderer/src/workspace/workspaceState.ts (lastJsonlEntryAt + ghosts field docstrings) * src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts (orphan sweep + ghost mint sites + TTL constant) * src/renderer/src/workspace/tile-tree/TileLeaf.tsx (call-site) * src/main/ghostJournal.ts (file-level block) * src/providers/claude/runtime/claudeSession.ts (sidecar opt-in) Also tightens the TTL constant comment to reflect that 30000ms is the right value (long enough that normal slow operations don't trip it, short enough that real stuck-mid-turn cases surface within roughly half a minute on resume), and removes references to a transient "60000ms defensive margin" idea that was discussed and dropped. No behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/design/README.md | 46 +++ docs/design/ghost-system.md | 268 ++++++++++++++++++ src/main/ghostJournal.ts | 7 + src/providers/claude/runtime/claudeSession.ts | 18 +- src/renderer/src/workspace/ghosts.ts | 12 +- .../workspace/hook/ipc/useIpcSubscriptions.ts | 51 +++- src/renderer/src/workspace/mergedEntries.ts | 16 ++ .../src/workspace/tile-tree/TileLeaf.tsx | 5 +- src/renderer/src/workspace/workspaceState.ts | 16 +- 9 files changed, 404 insertions(+), 35 deletions(-) create mode 100644 docs/design/README.md create mode 100644 docs/design/ghost-system.md diff --git a/docs/design/README.md b/docs/design/README.md new file mode 100644 index 000000000..d69e5fecf --- /dev/null +++ b/docs/design/README.md @@ -0,0 +1,46 @@ +# `docs/design/` — sources of truth for non-obvious system designs + +Files in this directory describe **what cc-shell's most complicated subsystems are and why they are shaped the way they are.** They are referenced by inline code comments and treated as authoritative. They are *not* changelogs, *not* running notes, *not* blow-by-blow histories of how a fix was derived. + +## DO NOT change these files lightly + +A design doc here gets touched **only when the actual system changes shape.** Renaming a constant, fixing a bug, or shipping a comment cleanup does not change the design — those land as code commits without a doc edit. + +You should change a file in `docs/design/` when: + +- A subsystem's responsibilities or invariants change. +- A new component enters or leaves the picture. +- A previously-load-bearing rule is replaced by a different one. +- A "deliberately out of scope" item is brought into scope. + +You should NOT change a file in `docs/design/` when: + +- Fixing a bug whose root cause is consistent with the documented design. +- Refactoring within the existing rules. +- Adjusting a tunable (a TTL, a threshold, a cap) that the doc already describes by purpose rather than by value. +- Rewording a comment for clarity. + +When in doubt: the doc is wrong if and only if its description no longer matches reality. If reality matches and the wording is just unclear, that's a code-comment problem, not a design-doc problem. + +## What goes here vs. elsewhere + +| Kind of doc | Lives in | +|---|---| +| Source-of-truth design explanation, evergreen | `docs/design/.md` (this directory) | +| Date-stamped plan, executed once | `docs/superpowers/plans/YYYY-MM-DD-.md` | +| Long-form diagnostic / forensic write-up of a specific incident | `docs/superpowers/plans/YYYY-MM-DD--findings.md` | +| Style or convention reference | `docs/-style.md` (root of `docs/`) | +| README for the repo / a package | `README.md` next to the code | + +If you are documenting *what is true today and is expected to stay true*, it belongs here. If you are documenting *what we did in a particular moment*, it belongs in `plans/`. + +## How to add a new file + +1. Pick a short, durable filename: one subsystem, one file. `ghost-system.md`, not `2026-05-07-ghost-fix.md`. +2. Write it in plain English first, then the technical rules. Aim for the level a senior engineer joining the team next month needs — not a domain expert, not a beginner. +3. End with a `## Warning` section explaining what care must be taken when modifying the subsystem, and what the load-bearing invariants are. The whole point of these docs is to make future-you stop and read before pulling a thread. +4. Reference the doc from inline comments at the top of every file involved in the subsystem. One line per file is enough: `// See docs/design/.md for the canonical explanation.` Don't repeat the doc's contents in code comments — code comments explain *this file's role* in the design, the doc explains *the design.* + +## Current files + +- [ghost-system.md](ghost-system.md) — The provisional-record bridge between the proxy semantic stream and the durable JSONL transcript. Covers the five-rule render predicate, reconciliation, the orphan TTL, the sidecar shape filter, and the dual-owner relationship with `SemanticStreamingTurn`. diff --git a/docs/design/ghost-system.md b/docs/design/ghost-system.md new file mode 100644 index 000000000..1860b0894 --- /dev/null +++ b/docs/design/ghost-system.md @@ -0,0 +1,268 @@ +# Ghost system + +> **Source-of-truth design doc.** Be careful changing this file — it explains *why* the ghost subsystem is shaped the way it is, and the load-bearing invariants downstream code relies on. If you are about to alter ghost minting, reconciliation, the render predicate, or the orphan TTL, **read the [Warning](#warning) section first.** Bug fixes that don't change the design don't change this file. + +## What ghost is, in plain English + +cc-shell talks to the agent (Claude Code, Codex) two ways at the same time: + +- It **reads the agent's authoritative transcript file** (Claude Code's `~/.claude/projects//.jsonl`, Codex's rollout JSONL). That file is the durable record of what happened — but it's written *lazily*. Claude Code batches transcript writes every 100 ms (10 ms for remote sessions); Codex queues writes through a tokio mpsc channel that drains on flush barriers. The file is structurally behind reality. +- It **observes the agent's network traffic via a per-session mitmproxy.** The proxy emits a flat semantic stream — `turn_started`, `text_delta`, `tool_input_delta`, `tool_input_finalized`, `block_completed`, `tool_result`, `turn_completed`, etc. — as the model's reply streams in. The proxy is roughly real-time. + +The proxy is ahead. The file catches up. Eventually they agree. + +A **ghost** is a provisional `ClaudeEntry` cc-shell mints from the proxy stream, tagged `_atp.origin = 'ghost'`, with a deterministic uuid (`g--`). It exists to bridge the gap between the two sources without producing duplicate-render bugs. When the authoritative JSONL entry lands, the ghost is *superseded* and dropped from the visible set. If it never lands, the ghost is *orphaned* and (sometimes — see the predicate) surfaces as the only record of what the proxy observed. + +The ghost primitive is general-purpose and lives in the standalone `agent-transcript-parser` package (`./packages/agent-transcript-parser/src/ghost.ts`, doc at `./packages/agent-transcript-parser/docs/ghost.md`). cc-shell's job is to wire it up: mint ghosts from semantic events, persist to disk, reconcile against JSONL, decide when to render. + +## Why ghost exists + +The naive alternatives all fail: + +| Approach | Failure | +|---|---| +| Render only the JSONL file. | Visible 100 ms+ delay between the model talking and the screen showing anything. Worse during slow batched flushes or large tool_results. | +| Render only the proxy stream. | No durable record. Closing cc-shell mid-turn loses the partial turn. Resume after crash has nothing to show. | +| Render BOTH unconditionally — JSONL above, proxy below. | The same sentence appears twice during the lag window. Documented historical bug on Codex (`docs/superpowers/plans/2026-04-20-rendering-fixes.md`). | + +The design idea was: one merged feed. Authoritative entries from the file, plus provisional ghost rows when the proxy is ahead, with automatic supersedure when the file catches up. The user sees one row that transitions seamlessly from "live preview" to "committed." + +That plan was never fully completed (see [Phase 3: deliberately out of scope](#phase-3-deliberately-out-of-scope)). What ships today is the *bookkeeping* half of the system — minting, reconciling, persisting — combined with a tightly-scoped render path that surfaces ghosts only in the rare case where they're actually needed: when JSONL has stalled past the proxy. + +## The two visible owners of the live turn + +This is the most-confusing part of the current architecture, and it's load-bearing for every rule below. + +In cc-shell today, the **live current turn** (the assistant's in-flight reply) is rendered by `SemanticStreamingTurn`, which reads `runtime.semantic.currentTurn` directly off the semantic reducer. **Not from ghosts.** See `src/renderer/src/features/feed/ui/semantic/StreamingTurn.tsx` and the call site at `src/renderer/src/features/feed/ui/Feed.tsx:912-916`. + +Ghosts exist *in parallel* with that live render. Every semantic event that updates `currentTurn` ALSO mints / updates a corresponding ghost via `ghostsFromSemanticTurn` in `src/renderer/src/workspace/ghosts.ts`. The two paths don't collide because the render predicate (below) explicitly excludes any ghost whose `turnId === currentTurn.turnId` — `SemanticStreamingTurn` owns that turn, ghosts can't surface for it. + +So the practical question for the ghost system is: **what is rendering ghost rows actually for, given that `SemanticStreamingTurn` covers the normal live-turn case?** + +Answer: a small set of fallback cases where `SemanticStreamingTurn` can't help. + +1. **JSONL stuck mid-turn.** The agent process gets wedged or its writer backlogs while the proxy keeps emitting events. `currentTurn` eventually clears (or doesn't), JSONL never lands. The ghost on disk is the only record of the turn. +2. **Crash + restart.** cc-shell exited mid-turn. Semantic state was in-memory and is gone. JSONL has a partial transcript that ends before the in-flight turn. The ghost log on disk preserves what the proxy saw; on resume it surfaces the lost partial turn. + +Both reduce to the same condition: **proxy state is past the JSONL tail with no recovery in sight.** The render predicate detects this structurally. + +## Architecture: components and roles + +The whole subsystem reads in one direction — semantic events flow in, the rendered feed flows out — but it touches a lot of files. Here is what each one does. + +``` + ┌────────────────────────────────────────────────┐ + │ ClaudeProxyAdapter / CodexResponsesAdapter │ + │ (in packages/claude-code-headless, │ + │ packages/codex-headless) │ + │ │ + │ Sees raw network events, publishes the │ + │ semantic stream. Filters known-bad sidecars │ + │ at source via isSidecarFlow. │ + └─────────────────────────┬───────────────────────┘ + │ semantic events + ▼ + ┌───────────────────────────────────────────────────────────────┐ + │ src/renderer/src/workspace/semantic/foldEvent.ts │ + │ Reduces semantic events into runtime.semantic.currentTurn. │ + │ Provider-aware turn ownership (Codex strict, Claude auto- │ + │ replace). Drops events for the wrong turnId. │ + └───────────────────────────────────────────────────────────────┘ + │ + ┌──────────────────────────┴─────────────────────┐ + ▼ ▼ + ┌────────────────────────────┐ ┌─────────────────────────────┐ + │ SemanticStreamingTurn │ │ src/renderer/src/workspace/ │ + │ (Feed.tsx mounts this) │ │ ghosts.ts │ + │ │ │ │ + │ Renders the live current │ │ ghostsFromSemanticTurn — │ + │ turn DIRECTLY off │ │ mints / updates ghosts from │ + │ runtime.semantic │ │ each semantic tick. │ + │ .currentTurn. │ │ │ + │ │ │ Reference-stable: returns │ + │ This is the everyday │ │ prev unchanged on no-op. │ + │ live-turn path. │ └────────────┬────────────────┘ + └────────────────────────────┘ │ + │ next ghost map + ▼ + ┌──────────────────────────────────────────────────────────────────┐ + │ src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts │ + │ │ + │ - Calls ghostsFromSemanticTurn at the semantic ingest site. │ + │ - Calls reconcileUpstream at the JSONL ingest site to supersede │ + │ ghosts whose authoritative entry just landed. │ + │ - Stamps lastJsonlEntryAt from the JSONL stream. │ + │ - Runs orphanStale every 1s with TTL=30s. │ + │ - Persists changed ghosts via window.api.ghostAppend. │ + └────────────┬──────────────────────────────────────┬──────────────┘ + │ │ + ▼ ▼ + ┌──────────────────────────┐ ┌──────────────────────────────┐ + │ src/main/ghostJournal.ts │ │ src/renderer/src/workspace/ │ + │ (main process) │ │ mergedEntries.ts │ + │ │ │ │ + │ One JSONL file per │ │ selectMergedEntries — the │ + │ session under │ │ five-rule render predicate. │ + │ /ghost-logs/. │ │ Decides which ghosts (if │ + │ Append-only, batched │ │ any) merge into the rendered │ + │ at 100ms. │ │ feed via mergeWithUpstream. │ + │ │ │ │ + │ Read on resume by │ │ Most ticks returns │ + │ src/renderer/src/ │ │ runtime.entries unchanged. │ + │ workspace/hook/ │ └──────────────┬───────────────┘ + │ actions/session.ts. │ │ + └──────────────────────────┘ ▼ + ┌─────────────────────────┐ + │ Feed.tsx renders this │ + │ as committed transcript │ + │ rows. │ + └─────────────────────────┘ +``` + +### Components + +- **atp ghost primitives** — `packages/agent-transcript-parser/src/ghost.ts`. Pure library: `createGhost`, `updateGhost`, `supersedeGhost`, `orphanGhost`, `reduceGhostLog`, `reduceGhostLogSansSuperseded`, `mergeWithUpstream`. No IO. Reusable outside cc-shell. +- **cc-shell renderer reducer** — `src/renderer/src/workspace/ghosts.ts`. Bridges semantic events to atp primitives: `ghostsFromSemanticTurn`, `reconcileUpstream`, `orphanStale`, `gcSupersededGhosts`, `ghostsToPersist`. Reference-stable on no-op. +- **render predicate** — `src/renderer/src/workspace/mergedEntries.ts`. The five-rule selector. Source of truth for "does this ghost surface in the feed." +- **IPC wiring** — `src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts`. The orchestration layer: mint on semantic, reconcile on JSONL, sweep orphans on a 1s timer, stamp `lastJsonlEntryAt` from JSONL bursts, persist diffs. +- **bootstrap on resume** — `src/renderer/src/workspace/hook/actions/session.ts` (ghost log read + reconcile against loaded JSONL tail) and `src/renderer/src/workspace/hook/actions/initialHistory.ts` (stamping `lastJsonlEntryAt` from the loaded tail). +- **disk persistence** — `src/main/ghostJournal.ts` (writer with batched 100 ms drain), `src/main/ipc/ghost.ts` (`ghost:append` / `ghost:read`), `src/preload/api/ghost.ts` (renderer bridge). +- **runtime field** — `src/renderer/src/workspace/workspaceState.ts` — `SessionRuntime.ghosts: Map` and `SessionRuntime.lastJsonlEntryAt: number | null`. +- **proxy-side sidecar filter** — `packages/claude-code-headless/src/proxy/ClaudeProxyAdapter.ts:isSidecarFlow`. Demotes Claude Code's auxiliary calls (title gen, branch-name, compaction summary, hook-agent) to `attribution: 'secondary'` so they don't open a turn at all. Catches most known sidecars but not the predict-next-prompt variant; the renderer-side shape filter (rule 5 below) is the backstop. +- **the parallel live owner** — `src/renderer/src/features/feed/ui/semantic/StreamingTurn.tsx`. Reads `runtime.semantic.currentTurn` directly. Not part of the ghost system but inseparable from it: rule 3 of the predicate exists specifically because `SemanticStreamingTurn` owns the live current turn. + +## Lifecycle + +``` +created ──► updated* ──► superseded + \──► orphaned +``` + +- **created** — `createGhost` mints a fresh ghost with a deterministic `(turnId, blockIndex)` uuid. `createdAt = updatedAt = now`. Append to the journal. +- **updated** — `updateGhost` produces a revised snapshot with new content and bumped `updatedAt`. Same uuid. Append to the journal. The reducer `reduceGhostLog` picks the freshest by `updatedAt` at read time. +- **superseded** — `supersedeGhost` sets `_atp.supersededBy = realUuid`. Triggered by `reconcileUpstream` when a JSONL entry lands that matches the ghost's `(turnId, blockIndex)` (or `tool_use_id`). Append to the journal as the final state record. +- **orphaned** — `orphanGhost` sets `_atp.orphanedAt`. Triggered by the TTL sweep (`orphanStale`, every 1 s, threshold 30 s). The ghost is now eligible for the render predicate's rule 2. + +A ghost is appended-only on disk. There is no in-place mutation, no file rewrite, no lock file. `reduceGhostLog` folds the append stream into a `Map` at read time. + +## The render predicate + +This is the source-of-truth rule for whether a ghost appears in the rendered feed. Implemented in `selectMergedEntries` (`src/renderer/src/workspace/mergedEntries.ts`). + +A ghost is **render-eligible** iff ALL FIVE of the following hold: + +1. **Not superseded.** `ghost._atp.supersededBy === undefined`. JSONL has not caught up; the ghost is still potentially the only record. +2. **Orphaned.** `ghost._atp.orphanedAt !== undefined`. The TTL has elapsed without a JSONL match. JSONL has had its chance for this ghost. +3. **Not the live current turn.** `ghost._atp.turnId !== currentTurnId`. `SemanticStreamingTurn` owns the live turn render; surfacing a ghost for the same turn would double-render. +4. **Past the JSONL tail.** `lastJsonlEntryAt !== null && ghost._atp.updatedAt > lastJsonlEntryAt`. The ghost was minted from a proxy event that occurred *after* the most recent JSONL entry this session has observed. This structurally distinguishes "JSONL stalled past the proxy" from "JSONL kept writing past this ghost." +5. **Not sidecar-shaped.** `!ghostHasSidecarShape(ghost)`. The shape predicate matches Claude Code's known auxiliary-call fingerprint: assistant role, single text block, ≤200 chars. This is the backstop for tail-sidecar leaks rule 4 cannot see. + +Why rules 4 AND 5, not just rule 4: the timestamp predicate is structurally correct for "stale orphan from earlier in the session, JSONL kept writing past it" — rule 4 catches that. But it cannot tell apart these two TAIL cases: + +- (a) JSONL stopped mid-turn, ghost has the lost partial turn — *should render.* +- (b) Last real JSONL entry committed at `t=100`, then a sidecar leak (predict-next-prompt / title-gen / branch-name) at `t=105`, no later real turn ever supersedes it — *should NOT render.* + +Both have `ghost.updatedAt > lastJsonlEntryAt`. Rule 5 is the structural shape check that picks (b) out of the tail. + +When all five rules pass, surviving ghosts are merged via `mergeWithUpstream` with `trustSupersededFlag: true`. Tail-append is correct here because by predicate-3 these ghosts are not the active turn, and by predicate-4 they are newer than every committed entry — they belong at the very end chronologically. + +When NO ghost passes, `selectMergedEntries` returns `runtime.entries` *by identity* (not a fresh copy). This is load-bearing for Feed's row memoization (see [Reference stability](#reference-stability)). + +### Trade-off in rule 5 (knowingly accepted) + +A real assistant turn that crashed before any JSONL write AND happened to be a single short text block (e.g. "Done.") would also be hidden — it matches the sidecar shape. Production data shows this is a rare invisible loss; the alternative (orphan title-gen / next-prompt fragments piling up at the bottom of every session) is a daily UX harm. Two prior fix attempts (`686b94e`, `2a83978`) traded along this axis; the current predicate keeps the shape filter as backstop. + +## Reconciliation matchers + +When a JSONL entry lands, `reconcileUpstream` walks the ghost map and supersedes anything whose authoritative record this entry is. Provider-aware: + +- **Claude:** assistant entries carry `message.id` which equals the `turnId` we used for the ghost. Match by `turnId === message.id`. One upstream entry can supersede every ghost block for that turn at once. +- **Codex:** rollout emits one entry per content block, each with its own uuid. The rollout response_id is stamped onto every mapped entry by `stampCodexTurnId` in `src/renderer/src/workspace/codex/rollout.ts`. Match by `(turnId === codexTurnId, blockIndex)`. +- **Both providers, fallback:** if the upstream entry has any `tool_use` block whose id matches a ghost's `_atp.context.toolUseId` or `context.callId`, supersede that specific ghost. This handles edge cases where turn ids don't line up (Codex rollout can mint fresh uuids on replay). + +Reconciliation is reference-stable: `reconcileUpstream` returns the input map unchanged when no match was found, so Feed's row memoization holds across no-op JSONL bursts. + +## Reference stability + +Three reducers MUST return their input unchanged on no-op: + +- `ghostsFromSemanticTurn` — many semantic ticks (`usage_updated`, redundant `block_started`, etc.) don't actually mutate any ghost. The reducer lazy-clones: returns `prev` directly until it has a real mutation to land. +- `reconcileUpstream` — most JSONL entries don't match any ghost (compact boundaries, system entries, entries whose ghosts already superseded earlier). The reducer returns `prev` when no match. +- `selectMergedEntries` — when no ghost passes the five-rule predicate, returns `runtime.entries` by identity, NOT a fresh `[...entries]`. + +This is not an optimization — it is **load-bearing for Feed's row memos.** If any of these reducers always allocates, every memoized row in Feed busts on every tick, and ReactMarkdown re-parses every text/code block from scratch. The pre-fix versions had this bug; performance was unusable. + +If you add a new code path that touches the ghost map, **uphold the contract**: clone only when you actually mutate, and return the input by identity when you didn't. + +## Persistence and resume + +The ghost journal lives at `/ghost-logs/.ghost.jsonl`. One file per session. Append-only JSONL. Owned by the main process; the renderer fires-and-forgets `window.api.ghostAppend`. + +Why a separate file and not the agent's own JSONL: cc-shell never writes into Claude Code's `~/.claude/projects//.jsonl` or Codex's rollout file. Those belong to the agent and are actively written by its batched queue; two writers on the same file is a torn-line / lost-write disaster. Ghosts are also a cc-shell-internal concern — no external tool needs them. + +Why batched at 100 ms: matches Claude Code's own transcript batch interval (`FLUSH_INTERVAL_MS` in `claude-code-src/full/utils/sessionStorage.ts`). Per-entry fsync during streaming would be tens of writes per second across every active pane. The trade-off — up to 100 ms of data loss on hard crash — is acceptable because ghost is provisional by definition and the agent's own JSONL survives independently. + +On resume (`src/renderer/src/workspace/hook/actions/session.ts:199-266`): + +1. After session spawn, fire-and-forget `window.api.ghostRead(sessionId)`. +2. Fold the returned entries via `reduceGhostLogSansSuperseded` (drops ghosts already superseded on disk so they don't resurface on a fresh session). +3. Merge into `runtime.ghosts` — disk ghosts only fill slots the live runtime hasn't already produced. If a uuid exists in-memory, prefer the in-memory copy because it's strictly fresher. +4. Run `reconcileUpstream` against `current.entries` (the JSONL tail loaded by `loadInitialHistoryForSession`). This catches the case where the previous session's ghost log has entries whose authoritative JSONL has already landed in the loaded tail — without this pass they'd resurface as orphans. +5. Persist any newly-produced supersede records. + +The `lastJsonlEntryAt` field is also primed on resume from the loaded tail's max `entry.timestamp` (see `initialHistory.ts`), so the predicate's rule 4 has the right anchor against ghost `_atp.updatedAt` from the previous run. + +## The orphan TTL + +`GHOST_ORPHAN_TTL_MS = 30000` (30 s). `GHOST_ORPHAN_SWEEP_MS = 1000` (sweep every second). + +A ghost is marked orphaned when `now - updatedAt > TTL` and it isn't already superseded or orphaned. The TTL is **not** "how fast we paint fallback" — it's "how long we wait before concluding JSONL isn't coming for this ghost." After the orphan flag fires, the ghost is merely *eligible* for the render predicate; rules 3, 4, 5 still gate visibility. + +In practice, during healthy operation the orphan flag rarely fires for a real assistant turn: + +- Active streaming bumps `updatedAt` on every `text_delta`, so the TTL never reaches threshold. +- Long tool execution freezes `updatedAt` on the tool_use ghost between `tool_input_finalized` and `tool_result`, but `currentTurn` stays alive across pending tools (`hasPendingSemanticTools` in `semantic/helpers.ts:158-175`), so rule 3 hides the ghost regardless of orphan state. + +The TTL does fire reliably in the genuine stuck cases: + +- **Live-stuck:** `currentTurn` cleared (or hangs), JSONL stalled, semantic stream went silent → orphan after 30 s, predicate evaluates, ghost may render. +- **Resume-after-crash:** ghost log loaded from disk, ghost's `updatedAt` is from before the crash → orphan sweep fires within the first second of resume → predicate evaluates against the loaded JSONL tail. + +## Phase 3: deliberately out of scope + +The original design intent was for ghost to be the **only** live render path. `SemanticStreamingTurn` would be deleted. The merged feed would render committed entries plus ghosts (in chronological position via ordered insertion in `mergeWithUpstream`), with seamless supersedure on JSONL catch-up. + +That deletion never happened. Deferring it is fine — `SemanticStreamingTurn` reliably handles the everyday live-turn case, and the gap that ghost rendering covers is genuinely small (just the JSONL-stalled-past-proxy fallback). The cost is the dual-owner relationship described in [The two visible owners of the live turn](#the-two-visible-owners-of-the-live-turn) and the extra rule 3 in the predicate. + +To do Phase 3, atp's `mergeWithUpstream` would need to learn ordered insertion (anchor by `parentUuid` / `turnId` / nearest committed neighbor instead of always tail-appending), and Feed would need to render the merged list as the only source of live blocks. Until that work happens, the predicate's rule 3 keeps the two paths from colliding. + +## Warning + +Be careful changing this subsystem. The five-rule predicate has been wrong four times in production: + +| Commit | What it tried | Why it regressed | +|---|---|---| +| `10e4fc5` (2026-04-20) | Hide all non-current-turn ghosts | Lost the legitimate orphan-fallback case entirely | +| `686b94e` (2026-04-24) | Render orphan ghosts as fallback after TTL | Couldn't tell stuck-mid-turn apart from sidecar leaks; orphan title-gen fragments piled up at feed tail forever | +| `2a83978` (2026-05-07) | Add 200-char shape filter on top of `686b94e` | Worked for short sidecars but missed predict-next-prompt with full conversation history | +| `fix/hide-orphan-ghost-tail` (abandoned local v1, 2026-05-07) | Hide all ghosts again | Killed the legitimate stuck-mid-turn fallback; user noticed | + +The current predicate (timestamp gate + shape filter) layered fix passed cross-review and lands in `fix/render-stuck-ghosts`. Each prior attempt looked correct in isolation. The lesson is that *any* simplification of the rules breaks at least one production case. Specifically: + +- **Don't drop rule 3** without first deleting `SemanticStreamingTurn`. The two surfaces will double-render the live turn. +- **Don't drop rule 4** by going back to a TTL-only orphan render. Stale-old-orphan-below-newer-rows comes back, plus every sidecar leak. +- **Don't drop rule 5** without first widening the proxy-side `isSidecarFlow` predicate to cover predict-next-prompt with full-history bodies. The tail-sidecar case is the dominant production failure mode. +- **Don't bump the orphan TTL aggressively** in either direction without thinking through what new code reads `orphanedAt`. Today the only consumer is the predicate; future code may treat orphan-ness as a stronger signal. +- **Don't lose the reference-stability invariants** in `ghostsFromSemanticTurn`, `reconcileUpstream`, or `selectMergedEntries`. Feed's row memos depend on them. Allocating a fresh map / array on no-op invalidates every memoized row in the feed. +- **Don't ghost tool RESULTS.** `blocksFromSemantic` deliberately drops `tool_result` / `function_call_output` / `custom_tool_call_output`. Their authoritative form arrives in upstream JSONL and synthesizing a provisional one would fabricate output the model never produced. Ghosting tool-call *inputs* is safe; ghosting outputs is not. +- **Don't change `entry.timestamp` semantics.** `lastJsonlEntryAt` is parsed from `entry.timestamp` (ISO 8601 string) on both Claude and Codex paths. The comparison against ghost `_atp.updatedAt` only works because both are wall-clock-when-the-producer-observed-the-event. If a future JSONL entry path stamps `Date.now()` instead of the real timestamp, rule 4 silently degrades on resume. + +## Pointers to deeper analysis + +If you need more than this doc: + +- `packages/agent-transcript-parser/docs/ghost.md` — the standalone library's design rationale for the ghost primitive. +- `docs/superpowers/plans/2026-05-07-ghost-system-findings.md` — long-form forensic write-up of the system, including the four prior fix attempts, evidence from production debug bundles, and the case table the predicate satisfies. Date-stamped because it's a snapshot in time, not an ongoing source of truth. +- `docs/superpowers/plans/2026-05-07-ghost-rendering-predicate.md` — the implementation plan that landed the current predicate. +- `docs/superpowers/plans/2026-04-20-rendering-fixes.md` — the original plan for the dual-render fix wave that introduced the ghost system. +- `docs/superpowers/plans/2026-04-18-headless-live-turn-redesign.md` — the original (un-finished) Phase 3 plan. diff --git a/src/main/ghostJournal.ts b/src/main/ghostJournal.ts index 96cd7682f..c35f2724f 100644 --- a/src/main/ghostJournal.ts +++ b/src/main/ghostJournal.ts @@ -6,6 +6,13 @@ // a cc-shell-owned JSONL file so the live view survives a crash, a // window reload, or an Electron restart. // +// See docs/design/ghost-system.md for the canonical explanation +// of the ghost subsystem — what it does, when ghosts surface in +// the rendered feed, and why this file exists at all. DO NOT +// change the persistence model (one file per session, append- +// only, 100 ms batch) without re-reading that doc; resume + the +// reduceGhostLog last-write-wins rule depend on it. +// // ----------------------------------------------------------------------------- // Why a separate file and not upstream's JSONL // ----------------------------------------------------------------------------- diff --git a/src/providers/claude/runtime/claudeSession.ts b/src/providers/claude/runtime/claudeSession.ts index eb5edbeaf..502d58e31 100644 --- a/src/providers/claude/runtime/claudeSession.ts +++ b/src/providers/claude/runtime/claudeSession.ts @@ -256,14 +256,16 @@ export class ClaudeSession extends EventEmitter { // that opted into proxy specifically want the richer stream. proxy: this.useProxy ? { - // Sidecar Haiku filtering — see ClaudeProxyAdapter for the - // full rationale. Without this, Claude Code's auxiliary - // Haiku calls (session title generation, compaction - // summaries, hook agents, teleport title-and-branch) leak - // into the visible transcript as orphan ghost entries - // because Claude Code never writes them to the JSONL - // rollout, so the renderer's ghost-supersede path can - // never fire. + // Sidecar Haiku filtering — see ClaudeProxyAdapter for + // the full rationale, and docs/design/ghost-system.md + // for how this interacts with the renderer-side shape + // filter (rule 5 of the predicate). Without this, Claude + // Code's auxiliary Haiku calls (session title generation, + // compaction summaries, hook agents, teleport + // title-and-branch) leak into the visible transcript as + // orphan ghost entries because Claude Code never writes + // them to the JSONL rollout, so the renderer's + // ghost-supersede path can never fire. // // We default to assuming the user is on a non-Haiku // primary model. Why this is OK as a default: diff --git a/src/renderer/src/workspace/ghosts.ts b/src/renderer/src/workspace/ghosts.ts index 982a6b9be..c0ca687a1 100644 --- a/src/renderer/src/workspace/ghosts.ts +++ b/src/renderer/src/workspace/ghosts.ts @@ -60,10 +60,16 @@ // a setRuntimes cascade that busted every useMemo([entries]) in // Feed via selectMergedEntries. // -// See `agent-transcript-parser/docs/ghost.md` for the underlying -// primitive's semantics, and +// See docs/design/ghost-system.md for the canonical explanation +// of the ghost subsystem — what it does, the five-rule render +// predicate, and the load-bearing invariants this file must +// uphold. DO NOT change the lifecycle, reconciliation, or +// reference-stability behavior here without re-reading that doc. +// +// See also `agent-transcript-parser/docs/ghost.md` for the +// underlying primitive's semantics, and // docs/superpowers/plans/2026-05-07-ghost-system-findings.md for -// the longer story of how cc-shell got here. +// the long-form diagnostic of how cc-shell got here. // // No function here performs IO. No function here subscribes to // events. Input goes in, new `Map` comes out. diff --git a/src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts b/src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts index 45c03bcc5..3b6f020af 100644 --- a/src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts +++ b/src/renderer/src/workspace/hook/ipc/useIpcSubscriptions.ts @@ -96,17 +96,30 @@ const WORKTREE_CACHE_TTL_MS = 5000 const WORK_CONTEXT_RECENT_RAW_LIMIT = 500 // Threshold before an unsuperseded ghost is marked orphaned. // -// 3000ms was the value used while orphan rendering was always-on -// (commit 686b94e) — the goal was to surface fallback fast. With -// the layered predicate in mergedEntries.ts an orphan flag merely -// makes a ghost ELIGIBLE for rendering; rules 4 (timestamp gate) -// and 5 (sidecar shape) still gate final visibility. So the TTL -// is now "how long do we wait before concluding JSONL had its -// chance for this ghost," not "how fast do we paint." 30000ms -// matches atp's library default and safely covers slow tool -// results (large Read, large Bash output, slow fetches) without -// prematurely qualifying a ghost mid-turn. Sweep cadence stays at -// 1s — that's the polling rate, not the threshold. +// History: 3000ms while orphan rendering was always-on (commit +// 686b94e), then 30000ms when the layered predicate landed. +// +// With the layered predicate in mergedEntries.ts, an orphan flag +// merely makes a ghost ELIGIBLE for rendering; rules 4 (timestamp +// gate) and 5 (sidecar shape) still gate final visibility, and +// rule 3 hides ghosts whose `turnId === currentTurnId`. So during +// healthy operation, even a tool that runs tens of seconds doesn't +// visibly orphan: the semantic reducer keeps `currentTurn` alive +// across pending tools (see hasPendingSemanticTools in +// semantic/helpers.ts), and rule 3 hides any ghost whose turn is +// still active. The TTL here is the failsafe boundary for "how +// long before we conclude JSONL had its chance" — reachable only +// when currentTurn has cleared and JSONL has genuinely stalled. +// +// 30000ms matches atp's library default and is the right balance: +// long enough that a normal slow operation doesn't trip it, short +// enough that a real stuck-mid-turn case surfaces within roughly +// half a minute on resume. +// +// Sweep cadence stays at 1s — that's the polling rate, not the +// threshold. +// +// See docs/design/ghost-system.md for the canonical explanation. const GHOST_ORPHAN_TTL_MS = 30000 const GHOST_ORPHAN_SWEEP_MS = 1000 @@ -220,6 +233,13 @@ export function useIpcSubscriptions( useEffect(() => { const worktreeCache = new Map() const recentWorkContextRawBySession = new Map() + // Ghost orphan sweep — see docs/design/ghost-system.md for the + // canonical explanation. Runs every GHOST_ORPHAN_SWEEP_MS, + // calls orphanStale to flag any ghost whose updatedAt has been + // silent for longer than GHOST_ORPHAN_TTL_MS, persists the + // resulting state changes to disk via ghostAppend. Reference- + // stable on no-op: orphanStale returns the input map by + // identity when no ghost crossed the threshold. const orphanSweepTimer = window.setInterval(() => { const now = Date.now() setRuntimes(prev => { @@ -709,7 +729,8 @@ export function useIpcSubscriptions( // the new semantic turn. This runs on every semantic tick; // `ghostsFromSemanticTurn` is idempotent and // reference-stable so no-op ticks (e.g. usage_updated - // events) do not churn the map. + // events) do not churn the map. See + // docs/design/ghost-system.md for the canonical explanation. // // WHY here and not inside `foldSemanticEvent`: // foldSemanticEvent is intentionally agnostic to @@ -718,9 +739,9 @@ export function useIpcSubscriptions( // outer runtime. The ghost map lives on SessionRuntime // because it needs to survive across semantic history // archival (when `currentTurn` flips to null) and because - // Phase 2 will persist it to disk with session-scoped - // file names. Calling the ghost reducer at this outer - // boundary keeps the layering clean. + // the ghost journal persists it to disk with session- + // scoped file names. Calling the ghost reducer at this + // outer boundary keeps the layering clean. const nextGhosts = ghostsFromSemanticTurn( nextSemantic.currentTurn, sessionId, diff --git a/src/renderer/src/workspace/mergedEntries.ts b/src/renderer/src/workspace/mergedEntries.ts index 900bd887f..27aaed1e5 100644 --- a/src/renderer/src/workspace/mergedEntries.ts +++ b/src/renderer/src/workspace/mergedEntries.ts @@ -85,6 +85,22 @@ // that ships, the live current turn stays owned by // SemanticStreamingTurn and ghost rendering is reserved for the // JSONL-stalled-past-proxy fallback case described above. +// +// ----------------------------------------------------------------------------- +// CANONICAL EXPLANATION +// ----------------------------------------------------------------------------- +// +// docs/design/ghost-system.md is the source of truth for the +// ghost subsystem. The five-rule predicate below is reproduced +// there with the full rationale; this file's comments reflect +// the predicate as implemented. If the rules need to change, +// update the doc first. +// +// Each prior simplification of these rules regressed in +// production (see the Warning section in the design doc for the +// commit-by-commit history). Changing this predicate without +// re-reading that doc is how cc-shell ends up with stale ghost +// rows at the bottom of every feed again. import type { Entry } from '@shared/types/transcript' import type { SessionRuntime } from '@renderer/workspace/workspaceState' diff --git a/src/renderer/src/workspace/tile-tree/TileLeaf.tsx b/src/renderer/src/workspace/tile-tree/TileLeaf.tsx index 47162c725..d68678fb4 100644 --- a/src/renderer/src/workspace/tile-tree/TileLeaf.tsx +++ b/src/renderer/src/workspace/tile-tree/TileLeaf.tsx @@ -341,8 +341,9 @@ export function TileLeaf({ // turn is owned by `SemanticStreamingTurn` below; the // `currentTurnId` argument hides any ghost for that // turn so the two surfaces never double-render. - // See ./mergedEntries.ts and - // docs/superpowers/plans/2026-05-07-ghost-system-findings.md. + // See docs/design/ghost-system.md for the canonical + // explanation of the predicate and the dual-owner + // model. entries={selectMergedEntries( runtime, runtime.semantic.currentTurn?.turnId ?? null, diff --git a/src/renderer/src/workspace/workspaceState.ts b/src/renderer/src/workspace/workspaceState.ts index 7b2e9edb6..3fe533ef1 100644 --- a/src/renderer/src/workspace/workspaceState.ts +++ b/src/renderer/src/workspace/workspaceState.ts @@ -403,23 +403,25 @@ export type SessionRuntime = { * proxy (render — proxy event is the only record) vs. a * sidecar leak Claude Code never logs to its rollout (hide — * JSONL kept writing real turns past it). See + * docs/design/ghost-system.md for the canonical explanation of + * the predicate this field feeds, and * docs/superpowers/plans/2026-05-07-ghost-system-findings.md - * for the full rationale. */ + * for the long-form diagnostic. */ lastJsonlEntryAt: number | null /** Ghost-record state keyed by ghost uuid (`g--`). * * Ghosts are provisional ClaudeEntry records emitted from the * live semantic reducer to paper over the gap between a * provider's streaming events and its durable JSONL write. See - * `./ghosts.ts` for the reducer and + * docs/design/ghost-system.md for the canonical explanation of + * the subsystem, `./ghosts.ts` for the reducer functions, and * `agent-transcript-parser/docs/ghost.md` for the underlying * primitive. * - * The Map is opaque to most code — Feed reads merged entries via - * the selector `selectMergedEntries`, and only the ghost reducer - * functions mutate this field. The in-memory shape is the only - * source of truth for Phase 1; Phase 2 adds disk persistence in - * `src/main/ghostJournal.ts` for crash recovery. */ + * The Map is opaque to most code — Feed reads merged entries + * via the selector `selectMergedEntries`, and only the ghost + * reducer functions mutate this field. Disk persistence lives + * in `src/main/ghostJournal.ts`. */ ghosts: Map } From 21dac2161790a952b1588602a692d2a08f5a067f Mon Sep 17 00:00:00 2001 From: Julius Olsson Date: Fri, 8 May 2026 10:04:10 +0200 Subject: [PATCH 6/6] docs(design): align ghost-system.md with the implementation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three discrepancies caught in cross-review: 1. Rule 4 of the predicate stated render-eligibility requires `lastJsonlEntryAt !== null && updatedAt > lastJsonlEntryAt`, but the code (mergedEntries.ts) deliberately lets a null lastJsonlEntryAt fall through and the test (test-ghost-fallback.ts case 9) asserts that behavior. The doc now reflects: the rule only gates when lastJsonlEntryAt is known; on a fresh session with no observed JSONL entry, the ghost falls through to rule 5. 2. The Codex reconciliation matcher was described as `(turnId === codexTurnId, blockIndex)`, but ghosts.ts matches by `turnId === codexTurnId` only — a single Codex entry landing supersedes every ghost in that turn (same shape as the Claude message-id contract). The doc now describes the matcher as it is implemented and notes that per-block disambiguation comes from the tool_use_id / call_id fallback. 3. The Live-stuck note read "currentTurn cleared (or hangs)" which was misleading: if currentTurn hangs alive with the same turnId, rule 3 hides the ghost and SemanticStreamingTurn continues to own visibility. The doc now says cleared, and adds a clarifying note that the hang case keeps the live owner in charge. No code change. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/design/ghost-system.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/design/ghost-system.md b/docs/design/ghost-system.md index 1860b0894..49209049f 100644 --- a/docs/design/ghost-system.md +++ b/docs/design/ghost-system.md @@ -154,7 +154,7 @@ A ghost is **render-eligible** iff ALL FIVE of the following hold: 1. **Not superseded.** `ghost._atp.supersededBy === undefined`. JSONL has not caught up; the ghost is still potentially the only record. 2. **Orphaned.** `ghost._atp.orphanedAt !== undefined`. The TTL has elapsed without a JSONL match. JSONL has had its chance for this ghost. 3. **Not the live current turn.** `ghost._atp.turnId !== currentTurnId`. `SemanticStreamingTurn` owns the live turn render; surfacing a ghost for the same turn would double-render. -4. **Past the JSONL tail.** `lastJsonlEntryAt !== null && ghost._atp.updatedAt > lastJsonlEntryAt`. The ghost was minted from a proxy event that occurred *after* the most recent JSONL entry this session has observed. This structurally distinguishes "JSONL stalled past the proxy" from "JSONL kept writing past this ghost." +4. **Past the JSONL tail (when known).** When `lastJsonlEntryAt !== null`, the ghost's `_atp.updatedAt` must be strictly greater than `lastJsonlEntryAt`. When `lastJsonlEntryAt === null` (fresh session, no JSONL entry has ever been observed), this rule does not gate — the ghost falls through to rule 5. The non-null branch structurally distinguishes "JSONL stalled past the proxy" (proxy event newer than every JSONL entry → render) from "JSONL kept writing past this ghost" (sidecar that real JSONL turns have moved past → hide). 5. **Not sidecar-shaped.** `!ghostHasSidecarShape(ghost)`. The shape predicate matches Claude Code's known auxiliary-call fingerprint: assistant role, single text block, ≤200 chars. This is the backstop for tail-sidecar leaks rule 4 cannot see. Why rules 4 AND 5, not just rule 4: the timestamp predicate is structurally correct for "stale orphan from earlier in the session, JSONL kept writing past it" — rule 4 catches that. But it cannot tell apart these two TAIL cases: @@ -177,7 +177,7 @@ A real assistant turn that crashed before any JSONL write AND happened to be a s When a JSONL entry lands, `reconcileUpstream` walks the ghost map and supersedes anything whose authoritative record this entry is. Provider-aware: - **Claude:** assistant entries carry `message.id` which equals the `turnId` we used for the ghost. Match by `turnId === message.id`. One upstream entry can supersede every ghost block for that turn at once. -- **Codex:** rollout emits one entry per content block, each with its own uuid. The rollout response_id is stamped onto every mapped entry by `stampCodexTurnId` in `src/renderer/src/workspace/codex/rollout.ts`. Match by `(turnId === codexTurnId, blockIndex)`. +- **Codex:** rollout emits one entry per content block, each with its own uuid. The rollout response_id is stamped onto every mapped entry by `stampCodexTurnId` in `src/renderer/src/workspace/codex/rollout.ts`. Match is by `turnId === codexTurnId` — same shape as Claude. A single landed Codex entry supersedes every ghost in that turn even though only one block's authoritative entry has arrived in this burst; subsequent block entries for the same turn arrive harmlessly because their ghosts are already superseded. Per-block disambiguation comes from the tool_use_id / call_id fallback below when a ghost needs to track a specific block by id. - **Both providers, fallback:** if the upstream entry has any `tool_use` block whose id matches a ghost's `_atp.context.toolUseId` or `context.callId`, supersede that specific ghost. This handles edge cases where turn ids don't line up (Codex rollout can mint fresh uuids on replay). Reconciliation is reference-stable: `reconcileUpstream` returns the input map unchanged when no match was found, so Feed's row memoization holds across no-op JSONL bursts. @@ -225,7 +225,7 @@ In practice, during healthy operation the orphan flag rarely fires for a real as The TTL does fire reliably in the genuine stuck cases: -- **Live-stuck:** `currentTurn` cleared (or hangs), JSONL stalled, semantic stream went silent → orphan after 30 s, predicate evaluates, ghost may render. +- **Live-stuck:** `currentTurn` cleared (without JSONL having caught up), semantic stream went silent → orphan after 30 s, predicate evaluates, ghost may render. Note: if `currentTurn` instead hangs alive with the same `turnId`, rule 3 keeps the ghost hidden and `SemanticStreamingTurn` continues to own visibility — the live-stuck branch only applies once the live owner has released the turn. - **Resume-after-crash:** ghost log loaded from disk, ghost's `updatedAt` is from before the crash → orphan sweep fires within the first second of resume → predicate evaluates against the loaded JSONL tail. ## Phase 3: deliberately out of scope