save_manifest seeds stale semantic_hash from prior manifest, masking LLM-omitted docs — the re-run promised by #1890 never happens #1948

Summary

When the model omits documents from an extraction response, the #1890 warning promises:

The model returned a response but omitted them; a re-run will retry them.

and _stamped_manifest_files (cli.py, #933) filters those files out of the dict passed to save_manifest so their semantic_hash stays empty and detect_incremental re-queues them.

That mechanism only works for files that have never been stamped. If a file was successfully extracted in any previous run, save_manifest seeds the new manifest from the existing one on disk and the file's old semantic_hash survives the omission. On the next incremental run the content hash still matches that inherited stamp, so the file is reported unchanged and is never retried. The #1890 promise is silently broken for exactly the runs where it matters most (--force re-extractions over an existing manifest).

Observed on 0.9.17 (also present at v8 HEAD, 0.9.18 dev): a graphify extract . --backend gemini --force over a 651-doc Obsidian vault ended with

[graphify] WARNING: 110/651 dispatched file(s) produced no nodes and are absent from the graph: … The model returned a response but omitted them; a re-run will retry them.

and the immediately following incremental run reported

0 docs changed; 683 unchanged; 0 re-extracted

Spot-checking one omitted file (AI/sessions/2026-06-12-….md, 0 nodes in graph.json) confirmed its graphify-out/manifest.json entry still carried the semantic_hash from an earlier successful run.

Root cause

save_manifest (graphify/detect.py) is merge-by-seed, not replace:

1. Seed loop (detect.py ~1557–1569): every row of the existing manifest whose file is still on disk and still in the scan corpus is copied into the new manifest verbatim, stale semantic_hash included. (This seeding is intentional and correct for subset-saving callers, save_manifest drops untouched files on incremental runs, forcing full re-extraction #917.)
2. Stamp loop (detect.py ~1578–1593): only iterates over the files dict the caller passed in. Files removed by _stamped_manifest_files (cli.py:57, called at cli.py ~2729) are not in it, so their seeded row is never overwritten or cleared.
3. detect_incremental(kind="semantic") then compares the file's current content hash against the inherited semantic_hash. The content hasn't changed since the old successful extraction, so it matches → unchanged → never re-queued.

So the #933/#1890 "leave it unstamped so it gets retried" contract holds only for first-ever extractions. Once a file has a stamp, every later omission (model drops it from a response, failed chunk on a --force run, …) is permanently masked.

Reproduction (deterministic, no LLM needed)

Simulates run 1 = successful extraction, run 2 = --force re-run where the model omits the doc (so _stamped_manifest_files filtered it out of the files dict), then checks whether the promised retry happens:

import json, sys, tempfile
from pathlib import Path
from graphify.detect import save_manifest, detect_incremental

tmp = Path(tempfile.mkdtemp()).resolve()   # resolve(): avoid macOS /var symlink aliasing
root = tmp / "vault"; root.mkdir()
doc = root / "doc.md"; doc.write_text("stable content that never changes\n")
manifest_path = str(tmp / "graphify-out" / "manifest.json")
corpus = {str(doc)}

# run 1: doc.md extracted successfully -> stamped
save_manifest({"document": [str(doc)]}, manifest_path, kind="both", root=root, scan_corpus=corpus)

# run 2 (--force): model omits doc.md -> _stamped_manifest_files drops it from the dict
save_manifest({"document": []}, manifest_path, kind="both", root=root, scan_corpus=corpus)

entry = next(iter(json.loads(Path(manifest_path).read_text()).values()))
print("semantic_hash after omission:", repr(entry["semantic_hash"]))   # expected: ''

inc = detect_incremental(root, manifest_path, kind="semantic")
requeued = [f for fl in inc["new_files"].values() for f in fl]
print("re-queued on next run:", requeued)                              # expected: [doc.md]

Output on v8 HEAD:

semantic_hash after omission: 'c561585056bce0a38112036a060bac99'
re-queued on next run: []

Expected: semantic_hash empty after run 2 and doc.md re-queued on the next incremental run, per the #1890 warning text and the #933 design.

Suggested fix direction

The seed loop can't just stop preserving stamps — that would break subset-saving callers (#917). The caller needs a way to say "these files were dispatched but produced no output; clear their semantic stamp":

• extract already knows the exact set — merged["uncovered_files"] is computed for the extract: LLM-omitted documents vanish silently — no dispatch/return reconciliation, no anti-regression guard (53% of docs lost even with gpt-5) #1890 warning (graphify/llm.py ~2081), and _stamped_manifest_files re-derives the same membership.
• Add e.g. a clear_semantic: set[str] | None parameter to save_manifest; seeded rows matching it (after the same root-resolution as manifest.json records only zero-node files, not the processed corpus #1897) get semantic_hash = "" instead of the inherited value. Full-pipeline callers in cli.py pass the uncovered/unstamped set; save_manifest drops untouched files on incremental runs, forcing full re-extraction #917 subset callers keep passing nothing and are unaffected.

Workaround

Manually blank semantic_hash in graphify-out/manifest.json for every source file with 0 nodes in graph.json, then run incremental extract — the 110 omitted docs were picked up and retried correctly after that.

Environment

• graphify 0.9.17 (bug also present on v8 HEAD / 0.9.18 dev cycle)
• macOS (Darwin 25.6.0), Python 3.x
• backend: gemini (but the masking is backend-independent; only the omission rate varies)