Repository navigation
fix(save): keyed positions, corrected header times, and a validator that can reject (#140, #139, #136, #119) - #160
Merged
Conversation
added 9 commits
July 25, 2026 23:36
last-save.json held one session and one line, so two live sessions overwrote each other: A saves, B saves, and A's next save no longer recognises its own ID, resumes from 0, and re-summarizes its whole span. The reporter saw one 99-exchange span summarized twice 2.5h apart, landing a near-identical triplet in the daily file. Two terminals, or a background/worktree session sharing the store, is all it takes. Positions are keyed by session now, newest 32 kept, evicted by last save so a long-running session is not dropped by newer ones while still live. session/line stay as a mirror of the most recent save, so a reader from a half-upgraded install degrades to the old behaviour rather than to nothing — and a pre-existing single-slot file is carried over instead of discarded, which would re-extract everything once per upgrade. The write goes through a temp file and a rename: the read-merge-write is not atomic, and a reader catching it mid-write would see truncated JSON and resume from 0 — the very duplicate being fixed. Both shell readers updated with it. session-start recovery now asks whether the previous session was EVER saved rather than whether it owns the one slot; the equality test force-saved an already-saved session whenever another had saved since. Co-Authored-By: Max <noreply>
#139 — the prompt injects {{TIME}} and says to copy it, but the model reads a transcript full of other timestamps and sometimes stamps one of those. The format check could not see it: a wrong time is still a well-formed one, so entries landed hours out of order and nothing warned. The time is the one field the pipeline knows better than the model, so it is overwritten rather than validated. A no-op when the model got it right. #136 — a response that is not an entry header is not an entry, but it was logged as a warning and appended anyway; a reporter found a permission prompt sitting in now.md. Memory is a summary of summaries, so a bad line does not fade — now.md rolls into today-*.md, then recent.md, then archive.md, each hop compressing it as though it were work. It is dropped now, but never destroyed: the text goes to tmp/rejected-*.md (last 20 kept) so nothing is lost, it just does not enter the compression chain. The position still advances, exactly as a SKIP does, or the span would be re-summarized on every run forever. Chose quarantine over the suggested features.strict_validation knob: a knob leaves the default broken for everyone who never finds it. #119 — the prompt-side cause. The only SKIP was for work already recorded, so a greeting, handoff summary or "what's next?" was faithfully summarized as the work done. There is a no-work SKIP rule now, and #136's rejection is the backstop for when it is ignored. Co-Authored-By: Max <noreply>
Review found the #139 fix corrupting the very entries it was meant to straighten. The format check only demands a space BEFORE the pipe, so "## 18:30 |main" passes it — and splitting the line on a literal " | " then found nothing to strip, leaving the whole original line as the "rest": "## 00:00 | ## 18:30 |main". A wrong-but-readable time became an unreadable header, which is worse than the bug being fixed. Split on the pipe itself, drop one leading space, and leave any later pipes to the branch name. The result is re-checked against the same regex before it is written, so a rewrite can never produce something the validator would have rejected. Two more from the same pass. session-start-hook asked jq whether the session key EXISTS, while both python readers require an int — a corrupt {"id": null} would have counted as saved there and resumed from 0 in the readers, re-summarizing the whole span. And the legacy branch of get_last_save_line handed back a null line verbatim, where every caller expects a number. Co-Authored-By: Max <noreply>
Review round 2. jq's `type == "number"` accepts a fractional value; python's `isinstance(v, int)` rejects it. And bool subclasses int in python, so a hand-edited `true` passed there as line 1 while jq rejected it. Either way the two readers disagreed, and disagreement is the whole failure: session-start skips the recovery save believing the span was captured, while the reader every later run uses resumes from 0 and re-summarizes it — the duplicate #140 exists to prevent, reached through a corrupt file instead of two live sessions. A shared _is_line_number helper on the python side excludes bools; the jq predicate excludes fractions. Neither can arise from cmd_save_position, which writes int() — this is the same hand-edited/truncated-file threat model the null and string checks already covered, finished properly. Co-Authored-By: Max <noreply>
Review round 3, and the previous commit picked the wrong side of its own question. jq cannot tell 1.0 from 1 — every JSON number is a double there, and `. == floor` is true for both — so rejecting integral floats in python while jq accepted them left exactly the disagreement the commit set out to remove: session-start calls the span saved and skips the recovery, while the reader every later run uses resumes from 0 and summarizes it again. jq cannot be made stricter here, so python matches jq: bools out, fractions out, integral floats in and coerced with int(). Checked across 28 shapes — negative, zero, -0.0, 1e3, huge, string, list, object, null — the two predicates now agree on every one. Also the call site the fix missed. _read_positions' legacy branch kept a bare isinstance(..., int), so a hand-edited `line: true` was merged into the keyed store as literal true, persisting the corruption instead of dropping it — while every reader treats that true as 0 and re-extracts. Three of four position checks had been updated; this is the fourth. It had no test at all, which is how it survived. Co-Authored-By: Max <noreply>
Round 4 found a fifth copy of the rule — post-tool-hook.sh's inline json still had a bare isinstance check, so it read 0 for a position every other reader resumed from. DELTA is computed from that, so the throttle that should fire a save after 50 new lines looked past-threshold on nearly every tool call for the life of a session; the cooldown was the only thing holding it back. Three rounds in a row found a copy that had missed an update, so patching copy five was not the fix. The hook now calls `pipeline.shell read-position`, which is the same code path everything else uses. That leaves jq as the only reimplementation, and it is pinned against the python one by the parametrized agreement test. Round 4 also found the 29th shape after I claimed 28 agreed: JSON has no infinity literal, but 1e400 overflows to one, and floor(infinite) == infinite — so jq called it a line number while python's is_integer() refused it. jq would then report the session saved and session-start would skip the recovery force-save, losing that session's tail. Guarded with isinfinite/isnan. Co-Authored-By: Max <noreply>
Round 5: the last commit claimed jq was the only reimplementation left. It was not. extract.py and shell.py each had their own python copy, and they had already drifted — shell.py's guarded against a top level that is not an object, extract.py's did not. So a last-save.json that was valid JSON but the wrong shape ([], null, 42 — a truncated write landing on a byte boundary that still parses, or a hand edit) raised AttributeError inside extract_session, which its except clause did not catch. That runs backgrounded, disowned, stderr discarded. The save died silently and kept dying on every later run while the file stayed that shape. Lost memory, nothing to notice it by. read_positions now lives in extract.py — the lower-level module, which shell.py already imports — and shell.py imports it rather than reimplementing it. One python reader, one jq reader pinned against it by the agreement test, and a test asserting shell.py has not grown a copy again. On the hot path: round 5 measured the read-position call ~23ms slower in isolation than the old inline heredoc, which is true. Measured through the whole hook it is 27ms FASTER — 456ms to 429ms — because the old code spawned python twice to answer one question. (My first attempt at that number said 24ms vs 452ms; the "old" hook was running from a scratch dir where resolve-paths.sh does not exist, so it exited at line 35 and measured nothing.) Co-Authored-By: Max <noreply>
Co-Authored-By: Max <noreply>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #140. Closes #139. Closes #136. Closes #119.
Four issues that all land on the same seam: what the summarizer returns, and what the pipeline is willing to believe about it.
#140 — one slot, two sessions
last-save.jsonheld one session and one line. Two live sessions — two terminals, or a background/worktree session sharing the store — overwrote each other: A saves, B saves, and A's next save no longer recognises its own ID, resumes from 0, and re-summarizes its whole span. The reporter saw one 99-exchange span summarized twice 2.5h apart, landing a near-identical triplet in the daily file, which the dedup context cannot catch once other entries have arrived in between.Demonstrated against the old code before touching it:
Positions are keyed by session now, newest 32 kept, evicted by last save rather than first so a long-running session is not dropped by newer ones while still live. The write goes through a temp file and a rename — the read-merge-write is not atomic, and a reader catching it mid-flight would see truncated JSON and resume from 0, which is the very duplicate being fixed.
Compatibility runs both directions, because an upgrade lands mid-session for real users: a pre-existing single-slot file is carried over rather than discarded, and
session/lineare still written as a mirror of the most recent save so a half-upgraded install degrades to the old behaviour instead of to nothing.Session-start recovery had the same flaw and is fixed with it — it now asks whether the previous session was ever saved, not whether it owns the one slot, which used to re-save an already-saved session whenever another had saved since.
#139 — the model stamping its own time
The prompt injects
{{TIME}}and says to copy it verbatim, but the model reads a transcript full of other timestamps and sometimes stamps one of those. The format check cannot see it, because a wrong time is still a well-formed one: daily files showed## 18:30written at 05:10 and## 18:15written at 06:08.The wall-clock time is the one field the pipeline knows better than the model, so it is written back over whatever came out — a no-op when the model copied it correctly, logged when it did not.
#136 / #119 — a validator that could only complain
The format validator detected a response that was not an entry header, logged a warning, and appended it anyway; one reporter found a permission prompt sitting in
now.md. That matters more than one junk line looks, because memory is a summary of summaries:now.mdrolls intotoday-*.md, thenrecent.md, thenarchive.md, each hop compressing it as though it were real work.Non-conforming output is now dropped from memory but not destroyed — it goes to
tmp/rejected-*.md(last 20 kept), so nothing is lost and nothing enters the compression chain. The position still advances exactly as a SKIP does, or the span would be re-summarized on every run forever.On the trade-off @oliver-mee raised: quarantine rather than a
features.strict_validationknob. A knob leaves the default broken for everyone who never discovers it, and the recoverable-file option means failing closed costs nothing.#119 is the prompt-side cause of the same pollution — the only SKIP rule covered work already recorded, so a greeting, handoff summary or "what's next?" was faithfully summarized as the work done. There is a no-work SKIP rule now, with the rejection above as its backstop.
What review caught
Two rounds, and the first found the #139 fix corrupting the entries it was meant to straighten:
Splitting on a literal
" | "found nothing to strip, so the whole original line became the "rest". A wrong-but-readable time became an unreadable duplicated header — worse than the bug. It now splits on the pipe itself, keeps later pipes for the branch name, and re-validates the assembled line against the same regex before writing, so a rewrite can never produce something the validator would reject.Round two found the two position readers disagreeing on corrupt input: jq's
type == "number"accepts a float, andisinstance(v, int)accepts a bool, since bool subclasses int. Disagreement is the whole failure mode — session-start skips the recovery save while the reader every later run uses resumes from 0.Tests
35 added, 637 passing. Each verified to fail on the commit before its fix.
Two are worth calling out because they nearly shipped green for the wrong reason:
now.mdpassed against unfixed code, because background NDC compression had already moved the junk intotoday-*.md. It checks every memory file now. Absence assertions are where this hides.get_last_save_linereturnedNonerather than0for a null line, to callers that expect an int.Noted, not fixed here
features.ndc_compressionis documented in the README but read nowhere in the code — setting it tofalsedoes nothing. That is what made the test above flaky. Filed as #159 rather than widening this PR.