Repository navigation
perf: accelerate MCP graph build/incremental updates and reduce resident memory (RSS) - #1139
3 commits merged into
Conversation
code-review-graph reviewOverall risk: 0.85 (CRITICAL) — 136 changed function(s)/class(es), 86 affected flow(s), 5 test gap(s) (2 with no tested caller found, 3 reached only through a caller) Risk-scored changes
Affected execution flows
Test gaps
Reached only through a callerNo test names these directly, but a tested caller reaches them along the call path shown. That is a path in the call graph, not a record of execution -- the caller's tests may never take this branch. Treat it as where to look, not as coverage: these count as gaps and are scored as untested.
Token savings: this graph-backed report used ~317,231 fewer tokens (~56%) than reading every changed file in full (estimated, chars/4 approximation). Powered by code-review-graph — local-first analysis; no code leaves the CI runner. |
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved rebuild handling can leave derived graph data stale, while an igraph import failure can replace existing communities with an empty set instead of using the fallback.
6 open issues
Fall back when igraph import fails after detection · New Recompute derived data when rebuild reports no changed files · New Keep temporary impact tables file-backed to bound memory · New Invalidate persistent context after graph mutations · New Exclude fallback-only languages from grammar prewarming · New Offload lazy imports and include them in the timeout · New
What changed in this PR
This PR targets faster graph builds and incremental updates, plus lower resident memory use in MCP servers and daemons.
Changes:
- Batches persistence and resolver queries, skips unchanged-graph work, and prewarms parser probes.
- Defers heavy imports, adds cache cleanup, and streams top-K embedding search.
- Adds direct helper tests and cleans up existing test formatting.
| File | Description |
|---|---|
| tests/test_transactions.py | Cleans up imports and formatting. |
| tests/test_prompts.py | Wraps a long assertion string. |
| tests/test_parser.py | Exercises cache cleanup and unreachable-call helpers. |
| tests/test_main.py | Exercises lazy attributes, offloading, and prompts. |
| tests/test_incremental.py | Exercises build, resolver, and daemon helpers. |
| tests/test_graph.py | Exercises batching, resolution, and memory cleanup. |
| tests/test_fts_sync.py | Cleans up imports and formatting. |
| tests/test_embeddings.py | Tests model-cache cleanup and score streaming. |
| tests/test_custom_languages.py | Exercises validation and lazy grammar imports. |
| tests/test_communities.py | Exercises cohesion and community-detection helpers. |
| code_review_graph/tools/build.py | Adds no-op handling and batches summary writes. |
| code_review_graph/tools/__init__.py | Lazily loads exported tool implementations. |
| code_review_graph/temporal_resolver.py | Limits call queries to Java files. |
| code_review_graph/spring_resolver.py | Limits call queries to Java files. |
| code_review_graph/scoped_resolver.py | Batches queries for relevant language files. |
| code_review_graph/rescript_resolver.py | Adds an early exit and batches updates. |
| code_review_graph/python_resolver.py | Reduces JSON parsing and adjusts update transactions. |
| code_review_graph/parser.py | Optimizes traversal, probe locking, and cache cleanup. |
| code_review_graph/main.py | Defers tool, prompt, and graph imports. |
| code_review_graph/incremental.py | Batches writes and optimizes dependency lookup and parsing. |
| code_review_graph/graph.py | Adds memory controls and caches resolution context. |
| code_review_graph/flows.py | Batches flow memberships and short-circuits entry checks. |
| code_review_graph/event_resolver.py | Narrows derived-event cleanup queries. |
| code_review_graph/embeddings.py | Streams top-K scores and adds memory cleanup. |
| code_review_graph/daemon.py | Reclaims memory after initial builds. |
| code_review_graph/custom_languages.py | Defers grammar-package imports. |
| code_review_graph/constants.py | Adds shared path and memory-reclamation helpers. |
| code_review_graph/communities.py | Defers igraph loading and uses lightweight edge records. |
| code_review_graph/analysis.py | Releases graph caches after bridge-node analysis. |
Review effort: Balanced
| global ig | ||
| if ig is None and IGRAPH_AVAILABLE: | ||
| try: | ||
| import igraph as _ig # type: ignore[import-untyped] | ||
|
|
||
| ig = _ig | ||
| except ImportError: | ||
| pass | ||
| if ig is None: | ||
| return [] |
There was a problem hiding this comment.
Fixed in 28f7bb7: _load_igraph() clears IGRAPH_AVAILABLE when the import raises, and every caller now falls back to file-based grouping for that call.
| use_incremental = ( | ||
| not full_rebuild | ||
| and changed_files is not None | ||
| and (bool(changed_files) or not build_result.get("stale_files_removed")) | ||
| ) |
There was a problem hiding this comment.
Fixed in 28f7bb7: use_incremental now requires a non-empty changed_files; an empty hint takes the full recompute path, and genuine no-ops stay on the existing branch.
| conn.execute("PRAGMA journal_mode=WAL") | ||
| conn.execute("PRAGMA busy_timeout=5000") | ||
| conn.execute("PRAGMA cache_size=-2000") | ||
| conn.execute("PRAGMA temp_store=MEMORY") |
There was a problem hiding this comment.
Fixed in 28f7bb7: PRAGMA temp_store=MEMORY is gone, so impact-radius working tables stay file-backed and shrink_memory can reclaim them.
| cached = getattr(self, "_bare_endpoint_context_cache", None) | ||
| if cached is not None: | ||
| return cached |
There was a problem hiding this comment.
Fixed in 28f7bb7: upsert_node, upsert_edge and commit now clear _bare_endpoint_context_cache, so resolution never reuses evidence a new edge invalidated.
| else: | ||
| grammars.add(lang) |
There was a problem hiding this comment.
Fixed in 28f7bb7: rescript, vbnet, yaml and properties are excluded from the prewarm set; a test asserts only the grammar-backed language is probed.
| impl_candidate = tool_name[:-5] if tool_name.endswith("_tool") else tool_name | ||
| for _cand in (impl_candidate, f"{impl_candidate}_func", tool_name): | ||
| if _cand in _LAZY_TOOL_NAMES and _cand not in globals(): | ||
| _resolve_attr(_cand) | ||
| break | ||
| if provenance and "with_provenance" not in globals(): | ||
| _resolve_attr("with_provenance") | ||
|
|
||
| run_fn = ( | ||
| (lambda: _resolve_attr("with_provenance")(work(), root)) | ||
| if provenance | ||
| else work | ||
| ) |
There was a problem hiding this comment.
Fixed in 28f7bb7: lazy resolution moved into the worker callable, so the cold import runs off-loop and inside the timeout.
28f7bb7 to
99fd19c
Compare
|
Addressed the review in 99fd19c. Review of that change surfaced three further defects, now fixed in the same commit:
Tests: the fallback, absolute-path handoff, cache drops and prewarm set each have a test verified to fail when its fix is reverted. Suites touched: 366 passed, 4 skipped. |
99fd19c to
23c1021
Compare
|
Extended in 23c1021 with the structural leftovers: the resolvers now commit through the store instead of the raw connection, so cache invalidation is no longer bypassed; a context fill that races an invalidation publishes only if the cache epoch is unchanged; custom languages are probed under their grammar name; and the networkx graph is built outside the cache lock and published under it. 316 passed, 4 skipped across graph, flows, incremental, communities and the resolvers. |
23c1021 to
3fca15a
Compare
|
Rebased onto staging (d72e8fb). The branch is three commits again (9c0f5de, 20eae3f, 3fca15a) and the tree is clean against ruff. The rebase dropped several parts of this PR because staging now carries an equivalent or better version:
What the branch still adds on top of staging:
Tests: 389 passed, 4 skipped across graph, flows, incremental, communities, main and the fallback tests; the weighted impact scoring suite passes 19/19. |
The build/RSS pass could leave the graph stale or block the event loop on six paths: - A lazy igraph import that failed after find_spec() succeeded kept the availability flag set, so community detection returned an empty set and replaced existing communities instead of using file-based grouping. - An empty change hint from the C++ identity rebuild took the incremental path, whose flow and community helpers return immediately for an empty list, so derived data stayed stale. - PRAGMA temp_store=MEMORY kept the impact-radius working tables resident after the delete, past the point where shrink_memory can reclaim them. - The bare-endpoint resolution cache was not cleared by upsert_node, upsert_edge or commit, so a later resolution reused evidence that a new IMPORTS_FROM edge had already invalidated. - Grammar prewarming probed rescript, vbnet, yaml and properties even though those languages are parsed by fallbacks and never load a grammar. - Lazy tool resolution ran inside the async wrapper before the worker was dispatched, blocking the loop on a cold import outside the timeout. Review of the first pass surfaced more: - The incremental helpers join on nodes.file_path, which is absolute, while update inputs are repository-relative, so every incremental flow and community pass matched nothing. The hint now reaches both helpers in the spelling the graph stores. - A broken native install raises OSError rather than ImportError, and an importable but empty package raises nothing at all; both escaped the fallback and failed the postprocess stage. The loader now handles both, logs the cause, and verifies the type it needs is present. - temp_store is pinned to FILE so the guarantee does not rest on the SQLite build default. - The resolvers committed through the raw connection, bypassing cache invalidation; they now commit through the store. rollback() drops the resolution context like commit() does. - A context fill that raced an invalidation could publish stale evidence; it now publishes only if the cache epoch is unchanged. - Custom languages are probed under their grammar name, not their alias. - The networkx graph is built outside the cache lock and published under it, so a long read no longer stalls writers. Tests cover the fallback, the absolute-path handoff, the cache drops, the prewarm set and the custom-language mapping; each was verified by reverting its fix and watching the test fail. Signed-off-by: Minxi Hou <houminxi@gmail.com>
3fca15a to
ae44bd2
Compare
|
The 3.10/3.11/3.12 failures after the rebase were mine, and they were real: |
…1139) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d01c1fd
…ew of tirth8205#1139) - The bare-call evidence context was cached until some write cleared it, but resolvers write through store._conn and skip that. A new import added that way was never seen: the next pass left the call bare. The call pass now always builds it fresh, and the tested-by pass drops it, so it never outlives the pair or holds every node name in a watch store. - GraphStore.close() and EmbeddingStore.close() ran gc.collect(). Every MCP tool call closes a store, and a full collection costs 10-15 ms in the server process, so it moved to the end of a build only. - EmbeddingStore set temp_store=MEMORY, the opposite of the graph fix. - Removed the batch writer leftovers: pending_batch was never filled and _can_batch_store was never called. - _single_hop_dependents and detect_communities sniffed whether a test had patched a GraphStore method. The dependents query is now GraphStore.get_direct_dependent_files, and the community edge scan always runs. - parser.py had its own copy of normalize_file_path; it re-exports the one in constants.py. - Restored the comments the PR deleted in the evidence builder. Each behaviour has a test in tests/test_derived_data_fallbacks.py that fails on the PR head. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tirth8205#1139) main() opens the watch store with GraphStore(...), but the PR imported GraphStore only under TYPE_CHECKING and served it through the module __getattr__, which a bare global lookup never reaches. The incremental module (and so graph.py) loads with the server anyway, so GraphStore and find_project_root are imported normally again; only tool and prompt modules stay deferred. The worker also guessed which implementation to load from the tool name. It now loads exactly the names the tool's lambda reads, and a test fails if a deferred name is read anywhere the worker does not resolve it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tirth8205#1139 skips flows and communities when an update stored no file. tirth8205#964 rebinds C# calls on a project-only change and marks the flows dirty, so the skip left them stale. A dirty mark now ends the skip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tirth8205#1139 loads the tool modules on first use. In test_cli.py an update test patches GraphStore and get_db_path while they load for the first time, so the mocks stayed bound in them and the real search test from tirth8205#823 that ran next found nothing. The test conftest now loads them up front. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tirth8205#1139 makes graph.py load the parser on first use, so EdgeInfo is only a type name there. tirth8205#715 builds EdgeInfo objects when it aligns edges with colliding names, which raised NameError on every store. The two methods now import it themselves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tirth8205#1139 loads igraph on first use through _load_igraph, so setting IGRAPH_AVAILABLE alone no longer makes detect_communities take the Leiden path when igraph is not installed. The tirth8205#882 test now stubs the loader too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


Summary
This PR resolves two major performance pain points when using
code-review-graphvia MCP and daemon mode:Key Optimizations
1. Graph Build & Incremental Update Speed (
incremental.py,tools/build.py,communities.py,flows.py,graph.py,parser.py)store_file_batch): Replaces per-file transactions (store_file_nodes_edges) duringfull_buildandincremental_updatewith 200-file chunked batch commits.find_dependents): UsesINDEXED BY idx_edges_target_kindwith batchedUNION + JOINSQL queries and reusesstore.get_file_hashes()instead of per-fileget_nodes_by_file()lookups (~1,530 ms -> ~50 ms ongraph.py)._run_postprocess&incremental_update: Skips 7 cross-file resolvers and full flow/community recomputation whenfiles_updated == 0and no stale files were removed, and fixeschanged_files=[]falsy fallback to full rebuild._prewarm_parser_probes): Pre-warms grammar probes with per-grammar locks before spawning worker pools.communities.py,flows.py): Replacesstore.get_all_edges()(which deserializedjson.loads(extra)80,886 times) with a lightweight tuple query and computes community cohesion in a singlecommunity_summaries,flow_snapshots,risk_index, andflow_membershipsinserts withexecutemany._bare_endpoint_context_cachebetweenresolve_bare_call_targetsandresolve_bare_tested_by_sources, filters cross-file resolvers by target language extensions, and adds a pure-strfast path tonormalize_file_path.2. Resident Memory (RSS) Reduction (
main.py,tools/__init__.py,graph.py,embeddings.py,parser.py,constants.py,daemon.py)main.py&tools/__init__.py: Defers loading heavy dependencies (networkx,igraph,tree_sitter_language_pack,parser.py) until a tool actually needs them, while preservingTYPE_CHECKINGannotations and backward-compatible module attributes.graph.py,embeddings.py): Caps SQLite page cache (PRAGMA cache_size=-2000, ~2 MB), guaranteesDROP TABLE IF EXISTSfor temporary impact-radius/unresolved-call tables infinallyblocks, and addsGraphStore.shrink_memory()+release_os_memory()(gc.collect()+ Linuxmalloc_trim(0)).parser.py,incremental.py): AddsCodeParser.clear_caches()and flushespending_batchevery 200 files so full-build peak RSS drops by over 35%.embeddings.py): Uses_iter_scores()generator withheapq.nlargestinstead of materializing full-table score lists, and addsclear_model_cache().Benchmark Results (378 files, 8,869 nodes, 81,885 edges)
Build & Incremental Speed
d72e8fb)ProcessPoolExecutor)12,449 ms5,689 msThreadPoolExecutor)29,956 ms15,393 ms3,848 – 4,288 ms106 – 117 msmain.py, 35 deps)3,174 ms813 msgraph.py, 186 deps)4,517 ms1,089 msResident & Peak Memory (
VmRSS/VmHWM)main.pyMCP Server Idle Startup (VmRSS)121.37 MB72.42 MBcode_review_graph.toolsCold Import (Delta RSS)+29.49 MB+1.19 MBGraphStoreOpen Instance (VmRSS)38.49 MB16.96 MBpostprocess='full') Peak Memory (VmHWM)164.83 MB106.63 MBVmRSS)144.38 MB83.13 MBVerification & Quality Gates
8,869nodes,81,885edges,139flows,19communities,8,803FTS5 rows).uvx ruff check code_review_graph/->All checks passed!(0 errors)uvx --with types-networkx mypy code_review_graph/ --ignore-missing-imports --no-strict-optional->Success: no issues found in 77 source files(0 errors)5045 passed, 0 failed).