[Bug]: --update stamps a partly-read doc as done, so it is not retried (update.md omits partial_source_files) #4807

Pre-flight checks

• I have checked the Troubleshooting section in the README

What happened?

When a subagent reads a doc only partway during /graphify --update, the skill says the doc will be retried on the next --update. It isn't. The --update runbook writes the manifest without the list of partly-read files, so the doc is stamped as done, and it stays half-extracted until its content changes again.

The partial-read handling from #2225 has three parts. Two of them run on every path:

• Step 3 B3.5 writes the doc to graphify-out/.graphify_partial.txt and prints "They are cached as partial and retried on the next --update" (skill.md:334).
• The semantic cache records it as partial, so the cache treats it as a miss (skill.md:381).

The third part keeps the doc unstamped in the manifest. Step 9 does this by passing partial_source_files=_partial (skill.md:662). But --update doesn't run Step 9: references/update.md says "Then run Steps 4–8" (update.md:179) and writes the manifest itself, without the partial set (update.md:164):

_manifest_files = _stamped_manifest_files(incremental['files'], new_extraction, Path('INPUT_PATH'))

So the doc is stamped, detect_incremental reads it as unchanged on the next run, and the cache miss is never reached. _stamped_manifest_files's docstring describes exactly this outcome: it would "never re-dispatch it, leaving the incomplete node set live forever on the warm-incremental path" (cli.py:323). The library's own paths pass the set (cli.py:418, cli.py:4789). The skill's --update runbook is the one path that doesn't.

The docs most exposed are long ones: a file over 2,000 lines is what the Read tool truncates without an error (#2225).

Expected: a doc read partway during --update stays unstamped and is re-dispatched on the next --update, as it already is after a full build.

Steps to reproduce

The script below replays the manifest write from references/update.md on a one-file corpus, once as shipped and once with the partial set that Step 9 passes. It needs no LLM; the extraction is a stub.

python3 repro.py            # update.md as shipped
python3 repro.py --partial  # with partial_source_files, as Step 9 passes it

"""A doc that is only partly read during --update is stamped as done and not retried.

Replays the manifest write in references/update.md (no partial_source_files) and the one in
SKILL.md Step 9 (with it), on a one-file corpus. Run: python3 repro.py; python3 repro.py --partial
"""

import os
import sys
import tempfile
from pathlib import Path

from graphify.cli import _stamped_manifest_files
from graphify.detect import detect, detect_incremental, save_manifest


def queued_docs() -> list[str]:
    return [Path(f).name for f in detect_incremental(Path("."))["new_files"].get("document", [])]


def run(pass_partial: bool) -> None:
    os.chdir(Path(tempfile.mkdtemp()).resolve())
    doc = Path("long.md").resolve()
    doc.write_text("".join(f"line {i} about Widget\n" for i in range(3000)))
    extraction = {"nodes": [{"id": "long_widget", "source_file": str(doc)}], "edges": [], "hyperedges": []}

    # Full build: the doc produced output and is stamped.
    save_manifest(_stamped_manifest_files(detect(Path("."))["files"], extraction, Path(".")), root=".")

    # The doc changes; --update dispatches it, the subagent reads 2000 of 3001 lines,
    # and Step 3 B3.5 writes it to .graphify_partial.txt.
    doc.write_text(doc.read_text() + "line 3000 about Sprocket\n")
    inc = detect_incremental(Path("."))
    assert queued_docs() == ["long.md"]

    # The manifest write from references/update.md, with or without the partial set.
    extra = {"partial_source_files": {str(doc)}} if pass_partial else {}
    manifest = _stamped_manifest_files(inc["files"], extraction, Path("."), **extra)
    dispatched = {f for t, fl in inc["new_files"].items() if t in ("document", "paper", "image") for f in fl}
    stamped = {f for fl in manifest.values() for f in fl}
    scan = {f for fl in inc["files"].values() for f in fl}
    save_manifest(manifest, root=".", scan_corpus=scan, clear_semantic=(dispatched - stamped) or None)

    label = "with partial_source_files" if pass_partial else "update.md as shipped     "
    print(f"{label}: next --update queues {queued_docs()}")

    # Only another edit brings it back.
    doc.write_text(doc.read_text() + "line 3001 about Gear\n")
    print(f"{label}: after another edit, queues {queued_docs()}")


run(pass_partial="--partial" in sys.argv)

Error output or graph output

update.md as shipped     : next --update queues []
update.md as shipped     : after another edit, queues ['long.md']
with partial_source_files: next --update queues ['long.md']
with partial_source_files: after another edit, queues ['long.md']

As shipped, the partly-read doc is not queued again until it is edited. With the partial set it is queued on the next run.

Graphify version

0.9.96. update.md is unchanged on v8 at dac0634.

Operating System

macOS

Python Version

3.15 (not in the dropdown)

Installation Method

pip

Additional Environment Details

No provider API keys set: this is the skill path, where the host agent's subagents do semantic extraction. The repro script calls no LLM.

Additional context

Update: #4816 implements this with one change. It deletes a leftover list at the start of --update instead of after the manifest save: an agent may run Step 9 after an --update (#837), and Step 9 reads the list too, so deleting it after the save gets the doc stamped there.

Suggested fix (revised; the original is below). In tools/skillgen/fragments/references/shared/update.md, read the partial list as Step 9 does, pass it on, and remove the list once the manifest is saved. Then regenerate with python -m tools.skillgen:

_pf = Path('graphify-out/.graphify_partial.txt')
_partial = {l.strip() for l in _pf.read_text(encoding=\"utf-8\").splitlines() if l.strip()} if _pf.exists() else set()
_manifest_files = _stamped_manifest_files(incremental['files'], new_extraction, Path('INPUT_PATH'), partial_source_files=_partial)
# ... unchanged ...
save_manifest(_manifest_files, root='INPUT_PATH', scan_corpus=_scan, clear_semantic=_cleared or None)
_pf.unlink(missing_ok=True)

The runbook's existing clear_semantic (dispatched minus stamped) then clears the doc's stale semantic_hash. The --partial run above exercises that.

The removal is needed too. --update never runs Step 9, which is what deletes the list after a full build, and Step B3.5 rewrites the list only when it runs. The code-only route skips it, and the all-cache-hit route says to run Step B3's commands without saying whether B3.5 runs. A list left over from an earlier run would then be read again. A doc it names that isn't dispatched keeps its manifest row, which is harmless. But a doc it names that is dispatched as a cache hit (its content reverted to a version cached in full) would be left unstamped and cleared on every run, so it would be re-queued on every --update. Today the runbook ignores the list, so this can't happen; passing the list without removing it would introduce it.

from graphify.cli import _stamped_manifest_files
_pf = Path('graphify-out/.graphify_partial.txt')
_partial = {l.strip() for l in _pf.read_text(encoding=\"utf-8\").splitlines() if l.strip()} if _pf.exists() else set()
_manifest_files = _stamped_manifest_files(incremental['files'], new_extraction, Path('INPUT_PATH'), partial_source_files=_partial)

Related: #2865 reorders this same block (stamp before build_merge), so the two fixes could land together.