Persist crash-adjacent console breadcrumbs and structured OOM observability events #370

Problem

The new App Run Journal captures a lot of crash context, but the recent OOMs still depended on terminal-only breadcrumbs that are not persisted anywhere durable.

Examples from the crash terminal output:

• [CodexResponsesAdapter] watchdog releasing proxy-* ...
• [debug-retention] pruned ... reason=performance-append ...
• Crash-adjacent stdout/stderr context around FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

Once the terminal scrollback is gone, those breadcrumbs are not present in events.jsonl, incidents.jsonl, heartbeat, heap snapshots, or debug bundles.

Related: #363, #364, #365, #368, #369.

Evidence

• CodexResponsesAdapter logs parse/watchdog errors and watchdog releases with console.warn, but it is constructed without an AppRunJournal sink. Its own proxy mirror persists upstream proxy events, not adapter watchdog decisions.
• debugRetention.ts logs structured prune summaries via console.info, but does not record them in the App Run Journal.
• AppRunJournal only persists explicit record / recordIncident calls.
• Debug bundles copy incident tails and heartbeat, but not a main-process console tail.
• Heartbeat is overwrite-only, so normal non-perf crash reports show only the last sample, not the growth slope leading into OOM.

Proposed Fix

Add durable, bounded crash-adjacent breadcrumbs:

1. Structured Codex proxy adapter events

   • Record watchdog releases and errors into AppRunJournal.
   • Include flowId, requestId, responseId, path, attribution, active flow id, silentForMs, flow count, turnOpened, buffered byte/string lengths, and whether finishTurn was synthesized.
   • Escalate repeated active-flow watchdog releases into an incident like codex.responses_adapter.stream_stalled.

2. Structured debug-retention events

   • Journal prune results instead of only printing them.
   • Include reason, removed, bytesFreed, scannedBytes, budgetBytes, ttlHours, elapsed time, and ideally per-bucket counts.

3. Bounded console/stdout/stderr tail

   • Add a capped console-tail.jsonl or wrap console.warn/info/error into a bounded AppRunJournal event sink.
   • Redact/truncate long payloads.
   • Never let the console tail become unbounded or block the process.
   • Include the tail in manual debug bundles.

4. Heartbeat history ring

   • Keep a tiny ring of recent samples, e.g. last 60, in addition to overwrite-only heartbeat.json.
   • Capture memory slope even when AGENT_CODE_PERF is off.

5. Prior-run incident enrichment

   • When recording app.prior_unclean_shutdown, include pointers to previous run heartbeat history, console tail, Crashpad minidump, heap snapshot, and proxy/debug directories when available.

Acceptance

• A future hard OOM can be diagnosed from retained files without relying on terminal scrollback.
• CodexResponsesAdapter watchdog releases are visible in the incident run artifacts.
• Debug-retention prune bursts are visible as structured events.
• The journal remains bounded under log storms.
• Manual debug bundles include the new crash-adjacent breadcrumb artifacts.