bug: --update shrinks manifest.json to changed-subset only, causing next --update to flag every unchanged file as new #837

Graphify version: 0.4.11 (pipx via miniforge), macOS 14, Python 3.12

Symptom

After a /graphify <path> (full rebuild), every subsequent /graphify --update reports a manifest that contains only the files seen in the prior --update run — not the cumulative corpus. The next --update then re-flags every other file in the corpus as "new", even though their content (and the existing graph.json) is unchanged.

Concrete reproducer in my corpus (a mixed wiki+code+raw archive, ~1028 files):

1. Full rebuild → manifest.json correctly contains all ~1028 files (Step 9 saves it from .graphify_detect.json).
2. Touch ~20 wiki entities; run /graphify --update. Behaves correctly for this run.
3. After the run, manifest.json now contains 24 entries (only the files touched in step 2's input, since --update's .graphify_detect.json is populated with new_files only).
4. Touch nothing; run /graphify --update again. detect_incremental reports 1008 new files — every file in the corpus except the 24 from step 2.

The content-hash semantic cache rescues this from being catastrophic — it hits on ~890 of the 1008 files and only ~118 are re-extracted. But:

• The "new files" report is misleading and triggers re-extraction of any uncached doc.
• The deletion-detection logic in the merge step has the wrong "manifest universe" to compare against, so it would miss true deletions.
• code_only fast-path doesn't fire when 99 stale code files appear in new_files.
• It's the #538 bug, but symmetric: that one fixed manifest persistence on the full-rebuild path; the --update path now persists too little.

Root cause

Two related defects in skill.md (installed/shipped Claude Code skill):

1. --update flow never populates .graphify_detect.json

The ## For --update section only writes .graphify_incremental.json, but Steps 3A–9 (which the section instructs the agent to "run as normal") all read from .graphify_detect.json. The skill works at all only because each agent ends up writing this file ad-hoc — and the natural ad-hoc shape is {'files': new_files}, which leads to defect 2.

2. Step 9 save_manifest(detect['files']) shrinks the manifest

# Step 9, current
detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text())
save_manifest(detect['files'])

save_manifest (graphify/detect.py) iterates the provided file list and writes {path: mtime} to manifest.json, overwriting the previous manifest. If detect['files'] is the --update subset (changed files only), every unchanged file's mtime is dropped from the manifest. Next detect_incremental sees stored_mtime is None for all of them → flags as new.

This is the same family as #538 (manifest not saved on full rebuild), but on the opposite path: here the manifest is saved on --update, but with the wrong (subset) data.

Suggested fix

Minimal, two-touch patch to graphify/skill.md:

(a) In the ## For --update section, after the detect_incremental block, add an explicit .graphify_detect.json population step that carries both keys:

$(cat graphify-out/.graphify_python) -c "
import json
from pathlib import Path
r = json.loads(Path('graphify-out/.graphify_incremental.json').read_text())
Path('graphify-out/.graphify_detect.json').write_text(json.dumps({
    'files': r.get('new_files', {}),           # drives processing in Steps 3A-3C
    'all_files': r.get('files', {}),           # full corpus, used by Step 9 save_manifest
    'total_files': r.get('new_total', 0),
    'total_words': r.get('total_words', 0),
    'skipped_sensitive': r.get('skipped_sensitive', []),
    'needs_graph': True,
}))
"

(b) Change Step 9's save_manifest(detect['files']) to prefer all_files when present:

# In --update mode, 'all_files' carries the full corpus; 'files' is the changed
# subset. Full-rebuild mode populates only 'files', so the fallback handles that.
save_manifest(detect.get('all_files') or detect['files'])

These changes are additive — full-rebuild flow is unaffected because it never writes all_files, and the fallback or detect['files'] preserves its existing semantics.

A PR is being prepared.

Observations / related

• Issue #826 (cache key path mismatch) is in the same operational area but distinct — the content-hash cache was masking some of this for me on subsequent runs.
• The defect mostly hides on code-only corpora because AST extraction is cheap and code_only would normally trigger the no-LLM fast-path. On mixed wiki/doc corpora it's much more visible: hundreds of doc files get flagged as "new" and the semantic cache becomes the only thing preventing a full re-extraction bill.
• I've already applied this patch locally and verified: manifest.json is restored to 1028 entries, and a subsequent detect_incremental reports new_total: 0 against an unchanged corpus.

Happy to revise the PR shape (e.g., move the file-population logic into detect.py instead of the skill markdown if that's preferred upstream).