Repository navigation
fix(skill): stop an --update re-read from stamping a doc whose chunk failed (#4811) - #4838
Deepjyoti-Sarmah wants to merge 7 commits into
Conversation
…bs#4814) A .graphify_partial.txt left by an --update (which stops before Step 9) is read by a later full build's Step 9, so a doc that build served in full is left unstamped and re-queued once. The A/B harness drives the real rendered B0/B3.5/B3-cache/Step-9 blocks against four candidate lifecycles - baseline, delete-at-start (A), run-id (B), freshness-on-read (C) - and scores them with the real detect_incremental and check_semantic_cache. Every policy starts from the same pre-fix counterfactual, so the matrix stays valid after the fix lands. Measured (see test_scoreboard): baseline 2/7, delete 7/7, runid 7/7, fresh 6/7 Choose A: B buys nothing on supported scenarios and adds a persistent file plus a header every reader must filter; C has a coarse-clock hole.
…run (Graphify-Labs#4814) Step B3.5 writes the list of files a subagent only partly read; Step 9 reads it and the same block deletes it. An --update stops at Step 8, so its list stays on disk. A later full build that serves that doc from the semantic cache never runs B3.5, so it never rewrites the list, and Step 9 reads the stale entry: the doc is left unstamped and the next --update re-queues a file this build already had in full. Apply the same rule B0 already applies to leftover .graphify_chunk_*.json files: delete the file before the stage that writes it. - B0 deletes it before B3.5 can rewrite it (full builds). - The --update runbook deletes it at its start, because an update stops before Step 9 and so can never rely on Step 9's cleanup. This also covers a code-only update, which skips B0 and B3 entirely. The A/B harness keeps reproducing the pre-fix baseline by stripping shipped hooks from its counterfactual, and gains an end-to-end regression against the real rendered fragments.
…aphify-Labs#4807) An --update stops before Step 9, so its only manifest save is the merge block in references/update.md, and that call passed no partial set. A doc B3.5 had just flagged as partly read therefore got stamped as done: detect_incremental never re-queued it and the head-only node set stayed live permanently - the same permanent-loss shape Graphify-Labs#2225 fixed on the full-build path. Pass the run's .graphify_partial.txt into _stamped_manifest_files, and drop the list after use because an --update has no Step 9 to clean up after it. Pinned by driving the real rendered manifest-save tail and asserting the doc is re-queued by detect_incremental and the list is gone afterwards.
) The Graphify-Labs#4807 regression extracted the manifest tail and ran it as Python, so it could not see a quoting error in the block's python -c wrapper. An unescaped encoding="utf-8" broke the block only when .graphify_partial.txt existed, which is exactly the path the test is about. Drive the shipped block through bash with the list on disk, and assert both the partial filter and the end-of-save cleanup.
An --update whose changed doc chunk fails leaves the doc queued, then the merge block overwrites .graphify_extract.json with the whole merged graph. Anything that re-reads that path as "what this run produced" stamps the doc: a second run of the merge block, or core Step 9 after the update (Graphify-Labs#837). The doc is then never re-queued and its graph keeps the pre-edit nodes. The tests drive the real rendered blocks through bash and read the verdict from the real detect_incremental, so they assert behaviour rather than the presence of a line. Twelve scenarios cover the two failure paths, partly-read docs, the success controls that must not start re-billing, the stale-state traps (a later full build, a second update), code-only and deletion-only updates, precision, and merge idempotency. A per-host test pins the hooks across all renders. Measured against the current runbook: 22 failed, 7 passed. The failures are exactly the stamped-after-re-read scenarios; every control passes.
…raphify-Labs#4811) The --update merge block reads this run's extraction from .graphify_extract.json, stamps the manifest from it, then overwrites the same file with the whole merged graph so Steps 4-8 can read it. After that the file no longer means "what this run produced": it is mostly old nodes from graph.json. Re-running the merge block in the same update therefore stamped a changed doc whose chunk failed, because its pre-edit node reappeared as fresh output. The doc was not re-queued and its graph kept the old content. Write a snapshot of this run's extraction, plus the partial set B3.5 wrote, before the merge runs, and stamp from the snapshot. The partial set rides along so the exclusion still holds after this block deletes .graphify_partial.txt (Graphify-Labs#4807/Graphify-Labs#4814). The snapshot is written only when absent, so a re-run keeps the first one instead of snapshotting the merged graph over it. Lifecycle, following the delete-before-the-writer rule the partial list already uses (Graphify-Labs#4814): the --update runbook deletes it at start, B0 deletes any leftover so a later full build cannot stamp from it, and Step 9's cleanup removes it. Step 9's own reader is a separate commit, so the two paths stay independently revertable. Its manifest-save tail test now composes the real pre-merge snapshot fragment with the real save fragment: the snapshot is part of that contract, and running the save alone would reference names the block defines earlier. Tests: 27 pass in tests/test_extract_snapshot_4811.py except the two [step9] scenarios, which the next commit turns green; the Graphify-Labs#4807 tail test and the no-backtick / lean-core guards pass. The lean core is still 799 lines, under its 800-line budget.
…y-Labs#4811) Step 9 read .graphify_extract.json and stamped every semantic file it found there. Run after an --update, that file holds the merged graph, so every doc in graph.json looked like this run's output: a changed doc whose chunk failed was stamped as done, never re-queued, and its pre-edit nodes stayed live. The same held for half-read docs once the update had consumed the partial list. Step 9 now prefers the snapshot written by the update merge block, which carries that run's extraction and partial set. A full build has no snapshot - B0 deletes any leftover - so it keeps stamping from .graphify_extract.json, which is this run's own output on that path. Both readers now read an immutable value, so nothing can replace it underneath them, and the partial exclusion survives the update's cleanup, so the Graphify-Labs#4807/Graphify-Labs#4814 protection holds on both paths. The lean core gains no net lines: the two new expressions replace the old one-line stamp call and its comment fits the budget. Tests: tests/test_extract_snapshot_4811.py 27 passed; the Graphify-Labs#4807 tail test, the no-backtick guard and the 800-line lean-core guard all pass.
|
Thanks for the pull request, @Deepjyoti-Sarmah. A maintainer will review it soon. Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions. A couple of things that speed up review: make sure the test suite passes on Python 3.10 and 3.13, and that the change keeps extraction deterministic. |
|
Thanks @Deepjyoti-Sarmah! One small thing: our changelog and release notes are built from PR titles, so we use the conventional-commits format, for example: Prefixes we use: |
There was a problem hiding this comment.
Graphify reviewed this change.
No issues found by static checks — no coupling regressions or blocking issues in the code graph. That is not the same as safe to merge: see what was not checked.
Not checked: tests were not run; no formal proof of the changed code.
Graphify review — findings
Fixes stale-state leakage between --update and later full builds, where a failed doc's old nodes could be stamped in the manifest as if they were fresh output (#4811). A full build now deletes .graphify_partial.txt and the previous .graphify_extract_run.json snapshot up front (#4814). Step 9 stamps the manifest from this run's pre-merge snapshot when one exists, falls back to .graphify_extract.json otherwise, and removes the snapshot during cleanup. The same change is applied to every platform skill file and the skillgen fragments and expected outputs, with tests covering the manifest tail.
Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 968 functions depend on the 966 functions this change touches.
Health — this change adds coupling hotspots:
- new:
_changed_doc_run()— 6 callers, 7 callees - new:
test_only_the_failed_doc_is_re_queued()— 0 callers, 10 callees - new:
_scenario_code_only_update()— 0 callers, 8 callees - new:
test_deletion_only_update_keeps_docs_stamped()— 0 callers, 7 callees - new:
test_the_shipped_runbook_drops_a_stale_list_before_step9()— 0 callers, 7 callees - new:
test_code_only_update_keeps_docs_stamped()— 0 callers, 6 callees - new:
_scenario_coarse_clock()— 0 callers, 6 callees - new:
test_a_half_read_doc_on_the_update_path_is_not_stamped()— 0 callers, 6 callees - …and 1 more — each is listed as a finding
Verification — 968 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 968 function(s) in the blast radius were not formally verified this run
Test selection
Test selection
471 of 471 test file(s) selected (100%) via static blast radius.
Escalated to a full run for safety — the selection is not trustworthy on its own (see below). CI should run the whole suite.
tests/pi.test.ts— full-run-safetytests/test_affected_cli.py— full-run-safetytests/test_affected_defines.py— full-run-safetytests/test_affected_member_seed.py— full-run-safetytests/test_affected_overrides.py— full-run-safetytests/test_affected_path_file_node.py— full-run-safetytests/test_agents_platform.py— full-run-safetytests/test_alias_folding.py— full-run-safetytests/test_analyze.py— full-run-safetytests/test_anthropic_custom_endpoint.py— full-run-safetytests/test_antigravity_install.py— full-run-safetytests/test_apm_fallback_version.py— full-run-safetytests/test_architecture_doc.py— full-run-safetytests/test_astro_extraction.py— full-run-safetytests/test_astro_import_ids.py— full-run-safetytests/test_atomic_canvas_export.py— full-run-safetytests/test_atomic_version_stamp.py— full-run-safetytests/test_atomic_writes.py— full-run-safetytests/test_backend_env_isolation.py— full-run-safetytests/test_backend_extras.py— full-run-safetytests/test_backend_opt_in.py— full-run-safetytests/test_benchmark.py— full-run-safetytests/test_benchmark_raw_graph.py— full-run-safetytests/test_benchmark_real_query.py— full-run-safetytests/test_binary_media_words.py— full-run-safetytests/test_blade_extractor.py— full-run-safetytests/test_build.py— full-run-safetytests/test_build_located_semantic_identity.py— full-run-safetytests/test_build_merge_dedup_scope.py— full-run-safetytests/test_build_merge_hyperedges_and_prune.py— full-run-safetytests/test_build_merge_shrink_guard.py— full-run-safetytests/test_builtin_global_type_refs.py— full-run-safetytests/test_c_include_resolution.py— full-run-safetytests/test_cache.py— full-run-safetytests/test_cache_long_paths.py— full-run-safetytests/test_cache_stale_import_target.py— full-run-safetytests/test_callflow_html.py— full-run-safetytests/test_cargo_introspect.py— full-run-safetytests/test_cargo_missing_manifest.py— full-run-safetytests/test_carried_hyperedge_remap.py— full-run-safetytests/test_case_sensitive_resolution.py— full-run-safetytests/test_charmap_encoding.py— full-run-safetytests/test_chunking.py— full-run-safetytests/test_cjs_module_extension.py— full-run-safetytests/test_claude_cli_backend.py— full-run-safetytests/test_claude_md.py— full-run-safetytests/test_cli_broken_pipe.py— full-run-safetytests/test_cli_export.py— full-run-safetytests/test_cli_help.py— full-run-safetytests/test_cloud_cta.py— full-run-safety- … and 421 more
non-code file(s) changed (
graphify/skill-agents.md,graphify/skill-amp.md,graphify/skill-claw.md,graphify/skill-codex.md,graphify/skill-copilot.md…) → running the full suite for safety (a code graph can't see config/fixture/data deps)
changed code file(s) with no mapped test (
graphify/skill-agents.md,graphify/skill-amp.md,graphify/skill-claw.md,graphify/skill-codex.md,graphify/skill-copilot.md…) — a coverage gap or a missing link — running the full suite rather than only the selected tests
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
Docs that may be stale (advisory)
ARCHITECTURE.md§ Pipeline (lines 5-12): references changed symbolsdetectARCHITECTURE.md§ Module responsibilities (lines 13-41): references changed symbolsdetectARCHITECTURE.md§ Adding a new language extractor (lines 87-94): references changed symbolsdetectBENCHMARKS.md§ Judge and grading (lines 68-84): references changed symbolspartialCHANGELOG.md§ 0.9.96 (2026-10-25) (lines 31-61): references changed symbolsdetectCHANGELOG.md§ 0.9.93 (2026-10-20) (lines 110-136): references changed symbolspartialCHANGELOG.md§ 0.9.88 (2026-10-12) (lines 209-233): references changed symbolsast,partialCHANGELOG.md§ 0.9.87 (2026-10-12) (lines 234-256): references changed symbolspartialCHANGELOG.md§ 0.9.86 (2026-10-11) (lines 257-278): references changed symbolsprojectCHANGELOG.md§ 0.9.84 (2026-10-10) (lines 302-315): references changed symbols__init__
…and 10 more.
· 9 grounded finding(s) anchored inline below.
| return proj | ||
|
|
||
|
|
||
| def _changed_doc_run(proj: Project, *, sem_nodes: list[dict], partial: bool) -> None: |
There was a problem hiding this comment.
_changed_doc_run()
fans out to 7 callees (efferent coupling); 6 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| assert "notes_widget2" in project.extract_node_ids(), "the retry's nodes never reached the graph" | ||
|
|
||
|
|
||
| def test_only_the_failed_doc_is_re_queued(project: Project): |
There was a problem hiding this comment.
test_only_the_failed_doc_is_re_queued()
fans out to 10 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| assert project.queue() == {"notes.md"}, "the successfully extracted doc came back too" | ||
|
|
||
|
|
||
| def test_code_only_update_keeps_docs_stamped(project: Project): |
There was a problem hiding this comment.
test_code_only_update_keeps_docs_stamped()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| assert project.queue() == set(), "a code-only update re-queued an untouched doc" | ||
|
|
||
|
|
||
| def test_deletion_only_update_keeps_docs_stamped(project: Project): |
There was a problem hiding this comment.
test_deletion_only_update_keeps_docs_stamped()
fans out to 7 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return project.next_update_queue() | ||
|
|
||
|
|
||
| def _scenario_code_only_update(project: Project) -> set[str]: |
There was a problem hiding this comment.
_scenario_code_only_update()
fans out to 8 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return project.next_update_queue() | ||
|
|
||
|
|
||
| def _scenario_coarse_clock(project: Project) -> set[str]: |
There was a problem hiding this comment.
_scenario_coarse_clock()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| ) | ||
|
|
||
|
|
||
| def test_the_shipped_runbook_drops_a_stale_list_before_step9(tmp_path): |
There was a problem hiding this comment.
test_the_shipped_runbook_drops_a_stale_list_before_step9()
fans out to 7 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return next(b for b in _bash_blocks(update_md) if "build_merge(" in b) | ||
|
|
||
|
|
||
| def test_a_half_read_doc_on_the_update_path_is_not_stamped(tmp_path): |
There was a problem hiding this comment.
test_a_half_read_doc_on_the_update_path_is_not_stamped()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| ) | ||
|
|
||
|
|
||
| def test_the_shipped_update_merge_block_runs_with_a_partial_list(tmp_path): |
There was a problem hiding this comment.
test_the_shipped_update_merge_block_runs_with_a_partial_list()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
What does this PR do?
Fixes #4811. An
--updatemarks a changed doc as done when its chunk failed, so the doc is never re-queued and the graph keeps its pre-edit nodes.The sequence: the merge block reads this run's extraction from
graphify-out/.graphify_extract.json, stamps the manifest from it (#2015/#1948), then overwrites the same file with the whole merged graph so Steps 4-8 can read it. From that moment the file no longer means "what this run produced" — it is mostly old nodes fromgraph.json, with nothing on disk saying which is which. Anything that re-reads it as this run's output stamps the doc:The fix writes a snapshot of this run's extraction, plus the partial set B3.5 wrote, before the merge overwrites the working file. The merge block's manifest save and Step 9 both stamp from that snapshot. The partial set travels with the payload, so the #4807/#4814 exclusion still holds after the merge block deletes
.graphify_partial.txt.Three commits, each independently revertable:
test(skill): pin the #4811 stamped-failed-doc gap— 12 scenarios over the real rendered blocks, red on the current runbook.fix(skill): snapshot this run's extraction before the update merge— the writer, the lifecycle, and the merge-block save. Turns the merge-rerun scenarios green.fix(skill): stamp Step 9 from this run's extraction snapshot— the second reader.Stacked on #4832 (
fix/4814-partial-list-lifecycle, currently open). That PR adds the run-start delete hooks this change extends, and its list-consumption line is the reason the partial set has to travel inside the snapshot. Until #4832 merges, this PR's diff includes its three commits.Design choice. I measured eight candidate designs against the same 12-scenario matrix (each driven through the real rendered blocks and scored with the real
detect_incremental):S1/S2 = failed chunk then merge re-run / Step 9. S3/S4 = partly read doc. S5/S6 = success controls. S7 = failed update then a clean full build. S8 code-only. S9 deletion-only. S10 full-build failed-chunk control. S11 second update after a failure. S12 precision.
Why the chosen design over the others:
Type of change
Verification & Invariants
The invariant: a file is stamped only when a value that describes this run says it produced output, and that value cannot be rewritten before every reader has read it. Step 9 and the merge block now read the same immutable snapshot; the snapshot is written once per run, before the merge, and cleared at every run start.
The issue's reproduction, before and after:
Generated state: all 134 skill artifacts are regenerated from the fragments and
--checkpasses. No graph, cache, or manifest format changes, so nothing to migrate.Merge Danger
Door: two-way. Runbook text, regenerated artifacts, one new test file, and one existing test helper. Reverting
972f14a5restores Step 9's old reader; reverting1fd842b9also removes the snapshot writer and lifecycle, returning the runbook to today's behaviour.bc036568is test-only (its own matrix is red by design, so it lands with the fixes).Blast Radius: skill-only, across the 14 hosts that render the shared core runbook.
Limitations:
references/update.md, so the snapshot is never written for them. They are out of scope, exactly as the [Bug]: --update stamps a partly-read doc as done, so it is not retried (update.md omits partial_source_files) #4807/[Bug]: a .graphify_partial.txt left by an --update can hold a cached doc back in a later full build #4814 hooks already are.graphify-out/remain unsafe; this change does not make the snapshot run-scoped beyond a fixed filename.--updateleaves its snapshot on disk until the next run's start hook or Step 9 cleanup, the same way the partial list already outlives an update.rm/Remove-Itemand Python lines that use single quotes only; the per-host rendered-artifact test covers every shared-core host.How was this tested?
The new test module drives the real rendered blocks through bash (shell wrapper included) and reads each verdict from the real
detect_incremental, so it fails on behaviour rather than on the presence of a line. It covers both readers, both failure modes, the success controls that must not start re-billing, the two stale-state traps, code-only and deletion-only updates, precision across two changed docs, merge idempotency, and a per-host check that every shared-core render carries the write/clean/read hooks. The full suite caught two issues during development, both fixed rather than worked around: a backtick in a comment inside apython -c "..."payload, and the lean-core 800-line guard (the Step 9 change is net-zero lines, so the guard is unchanged).Graphify-specific checklist
python -m tools.skillgen --bless) when changing their source fragments.test_skill_no_backtick_substitutionguard covers the backtick case.