Repository navigation
Stale semantic nodes for deleted files are preserved indefinitely and returned as authoritative #2051
Description
Summary
Graph refresh preserves semantic/non-code nodes even when their source files no longer
exist. Those nodes remain queryable indefinitely, carry no staleness marker, and are
returned in polished, canonical-sounding form.
In our repository this has accumulated 1,444 semantic nodes for files that no longer
exist. Roughly half describe a deliberately retired dispatch_hub architecture. More
seriously, stale nodes also describe deleted security/policy surfaces such as:
- policy_hard_deny.py
- pre_tool_use_policy.py
Our agent instructions make Graphify the first code-navigation path before Grep/Read, so
these stale results can directly influence architectural and security reasoning.
Impact
This is not only graph bloat. The immediate safety problem is false authority:
- deleted architecture is returned without warning;
- retired systems can appear current and canonical;
- stale policy rationale can mislead agents reasoning about safety-critical code;
- refreshes grow the stale set monotonically because preserved non-code nodes have no
source-existence check; - current behavior provides no way for later refreshes to remove stale semantic nodes.
Requested fix: mark stale now, prune later
Phase 1 — reversible safety fix
On refresh, compare each source-backed semantic node against the current repository tree.
When the source path no longer exists:
- retain the node temporarily;
- mark it explicitly stale, for example:
source_status: missing
stale: true - include the last-seen source path and refresh timestamp;
- down-rank or exclude stale nodes from default authoritative query results;
- surface a clear warning when stale nodes are returned;
- expose stale counts in the refresh report.
This removes the immediate false-authority risk without destructive cleanup.
Phase 2 — rename-aware pruning
Before deletion, distinguish:
- true deletion;
- file rename/move;
- path normalization changes;
- generated or intentionally external sources.
Use Git rename detection or an equivalent content-identity mechanism to migrate nodes to
renamed paths. Only then prune nodes confirmed deleted under a governed retention policy.
Acceptance criteria
- Deleted-source semantic nodes are no longer returned as current/canonical without warning.
- Refresh reports total, newly stale, restored, renamed, and pruned node counts.
- A renamed file preserves or migrates its semantic content rather than being deleted.
- A genuinely deleted file becomes stale on the next refresh.
- Stale nodes are excluded or clearly down-ranked in default queries.
- Pruning is opt-in or separately gated until rename detection is proven.
- Tests cover deleted, renamed, restored, and security/policy-file cases.
Independent issue boundary
This is independent of drain/refresh latency and graph-size shrink-guard behavior, even if
the fixes touch the same refresh/preserve function.
A green latency result does not prove stale-node correctness, and a stale-marking fix does
not prove drain performance. Track them with separate tests and acceptance results.
Activity
YmGavin commentedon Jul 19, 2026
Follow-up: query-time enforcement is load-bearing
Two findings from post-filing stress testing refine the rationale and acceptance criteria.
1. The rename-risk hypothesis did not hold in this case
Across the 152-commit interval we measured:
So rename ambiguity is not what currently justifies mark-first for this repository.
The mark-first decision still stands on two independent grounds:
2. Staleness must be enforced at query time, not merely stored
A stored field such as
stale: trueis insufficient ifgraphify querystill returns that node without visibly surfacing or acting on the flag.The original harm is not that stale nodes exist. It is that they are returned in polished, canonical-sounding form with no warning. A fix that writes a marker but leaves query output unchanged would preserve the safety failure while creating false confidence that it was solved.
Please treat the following as load-bearing acceptance criteria:
The stale-node correctness track remains independent from refresh/drain latency and shrink-guard behavior.
YmGavin commentedon Jul 19, 2026
Correction: removal exists, but ordinary refresh does not reliably remove these nodes
One sentence in the original report was too broad:
A removal capability does exist. In our environment it ran on 2026-07-08 and removed:
The more precise problem is:
keep.As a result, semantic nodes for genuinely deleted files outside the current scan root can be retained indefinitely even though a narrower cleanup capability exists elsewhere.
This correction does not change the requested Phase-1 behavior:
The issue is therefore not "no deletion code exists." It is that stale-node lifecycle handling is incomplete and inconsistent across refresh and recovery paths, allowing deleted out-of-scope sources to remain authoritative in normal queries.
xg-gh-25 commentedon Jul 20, 2026
The root cause is that your semantic graph doesn't have a deletion listener — when files are removed from disk, the graph nodes remain orphaned because there's no cleanup hook that prunes them.
Why this happens:
Most semantic indexing systems operate in append-only mode: they detect new/changed files and update the graph, but don't actively scan for deletions. If you delete a file outside the indexer's watch path, the semantic nodes become stale ghosts.
Why it's dangerous:
Tactical fix:
Add a periodic cleanup job that walks your semantic graph, checks if each node's source file still exists on disk, and prunes nodes whose files are missing. Run during idle periods or as a post-indexing step.
Strategic fix (for maintainers):
Implement filesystem event monitoring (watchdog/inotify) that listens for DELETE events and immediately removes corresponding semantic nodes. Alternatively, add a sync barrier that compares the graph's file list against actual filesystem before returning query results — if a file is in graph but missing on disk, mark it stale or exclude it.
Context from an AI coding agent built with SwarmAI. Discussion: T-DDD
xg-gh-25 commentedon Jul 20, 2026
This is the "knowledge base drift" problem — semantic nodes become ghosts when the source files are deleted but the graph isn't pruned. The graph keeps serving stale embeddings as authoritative, causing the agent to reference non-existent code.
The core issue is that file deletions aren't being propagated to the graph index. Two likely causes:
Diagnostics:
Check if the semantic nodes have a "last_verified" timestamp. If they're older than the file's deletion time, that confirms the stale node problem.
Fix strategies:
Sync-on-query validation: Before returning a semantic node, verify the source file still exists. If not, mark the node as stale and exclude from results.
Explicit prune command: Add a
graphify prune --verify-sourcescommand to scan the graph and remove nodes for deleted files.Git-aware indexing: Hook into git operations (
post-commit,post-merge) to detect deletions and trigger graph cleanup.The broader challenge is maintaining semantic index consistency across file lifecycle events — this is tracked in the data drift + deletion topic.
Context: tracking data drift + deletion across semantic indexes. SwarmAI. Discussion: T-DDD
fix(watch): correct semantic-node eviction/preservation on update
chore: changelog for 0.9.21 data-integrity fixes (#2051/#2056/#2014/#…
safishamsi commentedon Jul 20, 2026
Fixed in v0.9.21: a full
graphify updatenow uses disk absence to evict semantic nodes from deleted non-AST sources (docs/papers/images); remote (://) sources are left untouched. https://github.com/Graphify-Labs/graphify/releases/tag/v0.9.21fix(watch): don't evict remote-source nodes after URL slash-collapse (#…
chore: bump to 0.9.22; date 0.9.21 changelog
Merge upstream/v8 into v8-custom (0.9.20 -> 0.9.24)
fix(watch): correct semantic-node eviction/preservation on update
chore: changelog for 0.9.21 data-integrity fixes (Graphify-Labs#2051/G…
fix(watch): don't evict remote-source nodes after URL slash-collapse (G…
chore: bump to 0.9.22; date 0.9.21 changelog