Skip to content

perf: accelerate MCP graph build/incremental updates and reduce resident memory (RSS) - #1139

Merged
3 commits merged into
tirth8205:stagingfrom
HouMinXi:perf/mcp-build-speed-and-memory-rss
Nov 9, 2026
Merged

3 commits merged into
tirth8205:stagingfrom
HouMinXi:perf/mcp-build-speed-and-memory-rss

Conversation

@HouMinXi

Copy link
Copy Markdown
Contributor

Summary

This PR resolves two major performance pain points when using code-review-graph via MCP and daemon mode:

  1. Slow MCP graph building & incremental updates
  2. High resident memory (RSS) footprint during idle MCP server / daemon runtime and full builds

Key Optimizations

1. Graph Build & Incremental Update Speed (incremental.py, tools/build.py, communities.py, flows.py, graph.py, parser.py)

  • Chunked single-transaction batch persistence (store_file_batch): Replaces per-file transactions (store_file_nodes_edges) during full_build and incremental_update with 200-file chunked batch commits.
  • Eliminate N+1 dependent queries (find_dependents): Uses INDEXED BY idx_edges_target_kind with batched UNION + JOIN SQL queries and reuses store.get_file_hashes() instead of per-file get_nodes_by_file() lookups (~1,530 ms -> ~50 ms on graph.py).
  • Fast no-op short-circuit in _run_postprocess & incremental_update: Skips 7 cross-file resolvers and full flow/community recomputation when files_updated == 0 and no stale files were removed, and fixes changed_files=[] falsy fallback to full rebuild.
  • Concurrent Tree-sitter grammar probe pre-warming (_prewarm_parser_probes): Pre-warms grammar probes with per-grammar locks before spawning worker pools.
  • Lightweight edge scan & $O(E)$ community cohesion (communities.py, flows.py): Replaces store.get_all_edges() (which deserialized json.loads(extra) 80,886 times) with a lightweight tuple query and computes community cohesion in a single $O(E)$ pass instead of $O(C \times E)$. Batches community_summaries, flow_snapshots, risk_index, and flow_memberships inserts with executemany.
  • Shared bare-endpoint resolution context & fast path normalization: Shares _bare_endpoint_context_cache between resolve_bare_call_targets and resolve_bare_tested_by_sources, filters cross-file resolvers by target language extensions, and adds a pure-str fast path to normalize_file_path.

2. Resident Memory (RSS) Reduction (main.py, tools/__init__.py, graph.py, embeddings.py, parser.py, constants.py, daemon.py)

  • PEP 562 lazy loading in main.py & tools/__init__.py: Defers loading heavy dependencies (networkx, igraph, tree_sitter_language_pack, parser.py) until a tool actually needs them, while preserving TYPE_CHECKING annotations and backward-compatible module attributes.
  • SQLite cache & temp table bounds (graph.py, embeddings.py): Caps SQLite page cache (PRAGMA cache_size=-2000, ~2 MB), guarantees DROP TABLE IF EXISTS for temporary impact-radius/unresolved-call tables in finally blocks, and adds GraphStore.shrink_memory() + release_os_memory() (gc.collect() + Linux malloc_trim(0)).
  • Parser & AST cache lifecycle (parser.py, incremental.py): Adds CodeParser.clear_caches() and flushes pending_batch every 200 files so full-build peak RSS drops by over 35%.
  • Streaming Top-K vector search (embeddings.py): Uses _iter_scores() generator with heapq.nlargest instead of materializing full-table score lists, and adds clear_model_cache().

Benchmark Results (378 files, 8,869 nodes, 81,885 edges)

Build & Incremental Speed

Scenario Baseline (d72e8fb) Optimized Improvement
CLI Full Build + Full Postprocess (ProcessPoolExecutor) 12,449 ms 5,689 ms 2.19x faster (-54.3%)
MCP Non-TTY Full Build + Full Postprocess (ThreadPoolExecutor) 29,956 ms 15,393 ms 1.95x faster (-48.6%)
Incremental Update (no file change / short-circuit) 3,848 – 4,288 ms 106 – 117 ms 32.9x – 40.5x faster (-97%)
Incremental Update (single leaf file main.py, 35 deps) 3,174 ms 813 ms 3.91x faster (-74.4%)
Incremental Update (single core file graph.py, 186 deps) 4,517 ms 1,089 ms 4.15x faster (-75.9%)

Resident & Peak Memory (VmRSS / VmHWM)

Scenario Baseline Optimized Reduction
main.py MCP Server Idle Startup (VmRSS) 121.37 MB 72.42 MB -48.95 MB (-40.3%)
code_review_graph.tools Cold Import (Delta RSS) +29.49 MB +1.19 MB -28.30 MB (-96.0%)
GraphStore Open Instance (VmRSS) 38.49 MB 16.96 MB -21.53 MB (-55.9%)
Full Build (postprocess='full') Peak Memory (VmHWM) 164.83 MB 106.63 MB -58.20 MB (-35.3%)
Full Build + Postprocess + Bridge Nodes -> Close Store (VmRSS) 144.38 MB 83.13 MB -61.25 MB (-42.4%)

Verification & Quality Gates

  • Graph Equivalence: 100% identical outputs (8,869 nodes, 81,885 edges, 139 flows, 19 communities, 8,803 FTS5 rows).
  • Ruff: uvx ruff check code_review_graph/ -> All checks passed! (0 errors)
  • Mypy: uvx --with types-networkx mypy code_review_graph/ --ignore-missing-imports --no-strict-optional -> Success: no issues found in 77 source files (0 errors)
  • Pytest: Full test suite passed (5045 passed, 0 failed).

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

code-review-graph review

Overall risk: 0.85 (CRITICAL) — 136 changed function(s)/class(es), 86 affected flow(s), 5 test gap(s) (2 with no tested caller found, 3 reached only through a caller)

Risk-scored changes

Risk Level Symbol Location Tested
0.85 critical code_review_graph/custom_languages.py::_validate_entry code_review_graph/custom_languages.py:201 yes
0.85 critical code_review_graph/graph.py::GraphStore._invalidate_cache code_review_graph/graph.py:873 yes
0.75 high code_review_graph/graph.py::GraphStore.get_impact_radius_sql code_review_graph/graph.py:3063 yes
0.70 high code_review_graph/constants.py::normalize_file_path code_review_graph/constants.py:345 yes
0.65 medium code_review_graph/constants.py::release_os_memory code_review_graph/constants.py:368 yes
0.65 medium code_review_graph/graph.py::GraphStore.refresh_target_resolution code_review_graph/graph.py:2046 yes
0.65 medium code_review_graph/graph.py::GraphStore._get_bare_endpoint_context code_review_graph/graph.py:2345 yes
0.65 medium code_review_graph/graph.py::GraphStore._resolve_bare_endpoints code_review_graph/graph.py:2463 yes
0.65 medium code_review_graph/graph.py::GraphStore._build_networkx_graph code_review_graph/graph.py:3953 yes
0.65 medium code_review_graph/incremental.py::_get_code_parser_cls code_review_graph/incremental.py:40 yes

Affected execution flows

  • main — criticality 0.72, 1103 node(s) across 45 file(s)
  • benchmark_debug_workflow — criticality 0.70, 187 node(s) across 16 file(s)
  • run_token_benchmark — criticality 0.69, 57 node(s) across 6 file(s)
  • start_then_route — criticality 0.68, 24 node(s) across 5 file(s)
  • process — criticality 0.68, 142 node(s) across 16 file(s)
  • ...and 81 more affected flow(s)

Test gaps

  • code_review_graph/graph.py::GraphStore.rollback (code_review_graph/graph.py:1282)
  • code_review_graph/main.py::run_fn (code_review_graph/main.py:261)

Reached only through a caller

No test names these directly, but a tested caller reaches them along the call path shown. That is a path in the call graph, not a record of execution -- the caller's tests may never take this branch. Treat it as where to look, not as coverage: these count as gaps and are scored as untested.

  • code_review_graph/embeddings.py::EmbeddingStore._search_pure_python (code_review_graph/embeddings.py:1530) — via code_review_graph/embeddings.py::EmbeddingStore.search, 1 hop(s)
  • code_review_graph/graph.py::GraphStore._get_test_helper_routes (code_review_graph/graph.py:1626) — via code_review_graph/graph.py::GraphStore.get_caller_test_routes, 1 hop(s)
  • code_review_graph/hcl_resolver.py::resolve_hcl_module_references (code_review_graph/hcl_resolver.py:15) — via code_review_graph/incremental.py::full_build, 2 hop(s)

Token savings: this graph-backed report used ~317,231 fewer tokens (~56%) than reading every changed file in full (estimated, chars/4 approximation).


Powered by code-review-graph — local-first analysis; no code leaves the CI runner.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved rebuild handling can leave derived graph data stale, while an igraph import failure can replace existing communities with an empty set instead of using the fallback.

6 open issues
What changed in this PR

This PR targets faster graph builds and incremental updates, plus lower resident memory use in MCP servers and daemons.

Changes:

  • Batches persistence and resolver queries, skips unchanged-graph work, and prewarms parser probes.
  • Defers heavy imports, adds cache cleanup, and streams top-K embedding search.
  • Adds direct helper tests and cleans up existing test formatting.
File Description
tests/​test_transactions.py Cleans up imports and formatting.
tests/​test_prompts.py Wraps a long assertion string.
tests/​test_parser.py Exercises cache cleanup and unreachable-call helpers.
tests/​test_main.py Exercises lazy attributes, offloading, and prompts.
tests/​test_incremental.py Exercises build, resolver, and daemon helpers.
tests/​test_graph.py Exercises batching, resolution, and memory cleanup.
tests/​test_fts_sync.py Cleans up imports and formatting.
tests/​test_embeddings.py Tests model-cache cleanup and score streaming.
tests/​test_custom_languages.py Exercises validation and lazy grammar imports.
tests/​test_communities.py Exercises cohesion and community-detection helpers.
code_review_graph/​tools/​build.py Adds no-op handling and batches summary writes.
code_review_graph/​tools/​__init__.py Lazily loads exported tool implementations.
code_review_graph/​temporal_resolver.py Limits call queries to Java files.
code_review_graph/​spring_resolver.py Limits call queries to Java files.
code_review_graph/​scoped_resolver.py Batches queries for relevant language files.
code_review_graph/​rescript_resolver.py Adds an early exit and batches updates.
code_review_graph/​python_resolver.py Reduces JSON parsing and adjusts update transactions.
code_review_graph/​parser.py Optimizes traversal, probe locking, and cache cleanup.
code_review_graph/​main.py Defers tool, prompt, and graph imports.
code_review_graph/​incremental.py Batches writes and optimizes dependency lookup and parsing.
code_review_graph/​graph.py Adds memory controls and caches resolution context.
code_review_graph/​flows.py Batches flow memberships and short-circuits entry checks.
code_review_graph/​event_resolver.py Narrows derived-event cleanup queries.
code_review_graph/​embeddings.py Streams top-K scores and adds memory cleanup.
code_review_graph/​daemon.py Reclaims memory after initial builds.
code_review_graph/​custom_languages.py Defers grammar-package imports.
code_review_graph/​constants.py Adds shared path and memory-reclamation helpers.
code_review_graph/​communities.py Defers igraph loading and uses lightweight edge records.
code_review_graph/​analysis.py Releases graph caches after bridge-node analysis.

Review effort: Balanced

Comment thread code_review_graph/communities.py Outdated
Comment on lines 367 to 376
global ig
if ig is None and IGRAPH_AVAILABLE:
try:
import igraph as _ig # type: ignore[import-untyped]

ig = _ig
except ImportError:
pass
if ig is None:
return []

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 28f7bb7: _load_igraph() clears IGRAPH_AVAILABLE when the import raises, and every caller now falls back to file-based grouping for that call.

Comment on lines +309 to +313
use_incremental = (
not full_rebuild
and changed_files is not None
and (bool(changed_files) or not build_result.get("stale_files_removed"))
)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 28f7bb7: use_incremental now requires a non-empty changed_files; an empty hint takes the full recompute path, and genuine no-ops stay on the existing branch.

Comment thread code_review_graph/graph.py Outdated
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA busy_timeout=5000")
conn.execute("PRAGMA cache_size=-2000")
conn.execute("PRAGMA temp_store=MEMORY")

@HouMinXi HouMinXi Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 28f7bb7: PRAGMA temp_store=MEMORY is gone, so impact-radius working tables stay file-backed and shrink_memory can reclaim them.

Comment on lines +2157 to +2159
cached = getattr(self, "_bare_endpoint_context_cache", None)
if cached is not None:
return cached

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 28f7bb7: upsert_node, upsert_edge and commit now clear _bare_endpoint_context_cache, so resolution never reuses evidence a new edge invalidated.

Comment on lines +1735 to +1736
else:
grammars.add(lang)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 28f7bb7: rescript, vbnet, yaml and properties are excluded from the prewarm set; a test asserts only the grammar-backed language is probed.

Comment thread code_review_graph/main.py Outdated
Comment on lines +258 to +270
impl_candidate = tool_name[:-5] if tool_name.endswith("_tool") else tool_name
for _cand in (impl_candidate, f"{impl_candidate}_func", tool_name):
if _cand in _LAZY_TOOL_NAMES and _cand not in globals():
_resolve_attr(_cand)
break
if provenance and "with_provenance" not in globals():
_resolve_attr("with_provenance")

run_fn = (
(lambda: _resolve_attr("with_provenance")(work(), root))
if provenance
else work
)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 28f7bb7: lazy resolution moved into the worker callable, so the cold import runs off-loop and inside the timeout.

@HouMinXi
HouMinXi force-pushed the perf/mcp-build-speed-and-memory-rss branch from 28f7bb7 to 99fd19c Compare November 5, 2026 18:25
@HouMinXi

HouMinXi commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the review in 99fd19c. Review of that change surfaced three further defects, now fixed in the same commit:

  • The incremental helpers join on nodes.file_path, which is absolute, while update inputs are repository-relative. Every incremental flow and community pass therefore matched nothing. The hint now reaches both helpers in the spelling the graph stores.
  • A broken native install raises OSError, not ImportError, and an importable-but-empty package raises nothing at all. Both escaped the fallback and failed the postprocess stage; the loader now handles both, logs the cause, and checks that the type it needs exists.
  • PRAGMA temp_store is pinned to FILE so the guarantee does not depend on the SQLite build default, and rollback() drops the resolution context like commit() does.

Tests: the fallback, absolute-path handoff, cache drops and prewarm set each have a test verified to fail when its fix is reverted. Suites touched: 366 passed, 4 skipped.

@HouMinXi
HouMinXi force-pushed the perf/mcp-build-speed-and-memory-rss branch from 99fd19c to 23c1021 Compare November 6, 2026 12:32
@HouMinXi

HouMinXi commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Extended in 23c1021 with the structural leftovers: the resolvers now commit through the store instead of the raw connection, so cache invalidation is no longer bypassed; a context fill that races an invalidation publishes only if the cache epoch is unchanged; custom languages are probed under their grammar name; and the networkx graph is built outside the cache lock and published under it. 316 passed, 4 skipped across graph, flows, incremental, communities and the resolvers.

@HouMinXi
HouMinXi force-pushed the perf/mcp-build-speed-and-memory-rss branch from 23c1021 to 3fca15a Compare November 7, 2026 16:12
@HouMinXi

HouMinXi commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto staging (d72e8fb). The branch is three commits again (9c0f5de, 20eae3f, 3fca15a) and the tree is clean against ruff.

The rebase dropped several parts of this PR because staging now carries an equivalent or better version:

What the branch still adds on top of staging:

  • igraph fallback that also survives OSError and an importable-but-empty package, with the availability flag honoured.
  • temp_store=FILE pinned rather than left to the SQLite build default.
  • Cache invalidation on upsert_node, upsert_edge and commit; resolvers commit through the store; a fill that races an invalidation does not publish.
  • Prewarm skips languages with no bundled grammar and probes custom languages under their grammar name.
  • identity_rebuild never takes the incremental branch, and lazy tool resolution runs inside the worker.

Tests: 389 passed, 4 skipped across graph, flows, incremental, communities, main and the fallback tests; the weighted impact scoring suite passes 19/19.

The build/RSS pass could leave the graph stale or block the event loop on
six paths:

- A lazy igraph import that failed after find_spec() succeeded kept the
  availability flag set, so community detection returned an empty set and
  replaced existing communities instead of using file-based grouping.
- An empty change hint from the C++ identity rebuild took the incremental
  path, whose flow and community helpers return immediately for an empty
  list, so derived data stayed stale.
- PRAGMA temp_store=MEMORY kept the impact-radius working tables resident
  after the delete, past the point where shrink_memory can reclaim them.
- The bare-endpoint resolution cache was not cleared by upsert_node,
  upsert_edge or commit, so a later resolution reused evidence that a new
  IMPORTS_FROM edge had already invalidated.
- Grammar prewarming probed rescript, vbnet, yaml and properties even
  though those languages are parsed by fallbacks and never load a grammar.
- Lazy tool resolution ran inside the async wrapper before the worker was
  dispatched, blocking the loop on a cold import outside the timeout.

Review of the first pass surfaced more:

- The incremental helpers join on nodes.file_path, which is absolute,
  while update inputs are repository-relative, so every incremental flow
  and community pass matched nothing. The hint now reaches both helpers in
  the spelling the graph stores.
- A broken native install raises OSError rather than ImportError, and an
  importable but empty package raises nothing at all; both escaped the
  fallback and failed the postprocess stage. The loader now handles both,
  logs the cause, and verifies the type it needs is present.
- temp_store is pinned to FILE so the guarantee does not rest on the
  SQLite build default.
- The resolvers committed through the raw connection, bypassing cache
  invalidation; they now commit through the store. rollback() drops the
  resolution context like commit() does.
- A context fill that raced an invalidation could publish stale evidence;
  it now publishes only if the cache epoch is unchanged.
- Custom languages are probed under their grammar name, not their alias.
- The networkx graph is built outside the cache lock and published under
  it, so a long read no longer stalls writers.

Tests cover the fallback, the absolute-path handoff, the cache drops, the
prewarm set and the custom-language mapping; each was verified by
reverting its fix and watching the test fail.

Signed-off-by: Minxi Hou <houminxi@gmail.com>
@HouMinXi
HouMinXi force-pushed the perf/mcp-build-speed-and-memory-rss branch from 3fca15a to ae44bd2 Compare November 7, 2026 16:59
@HouMinXi

HouMinXi commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

The 3.10/3.11/3.12 failures after the rebase were mine, and they were real: _load_igraph checked hasattr(module, "Graph") outside the handler, so a native build that imports and then raises OSError on first attribute access escaped on those interpreters. The type is now touched inside the same try with AttributeError added to the handler. Reproduced on 3.12 locally (OSError: libigraph.so: cannot open shared object file), fixed, and the same six suites pass there: 389 passed, 4 skipped. ae44bd2.

tirth8205 added a commit that referenced this pull request Nov 9, 2026
…1139)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tirth8205 tirth8205 closed this pull request by merging all changes into tirth8205:staging in d01c1fd Nov 9, 2026
HengYangDS pushed a commit to HengYangDS/code-review-graph that referenced this pull request Nov 9, 2026
…ew of tirth8205#1139)

- The bare-call evidence context was cached until some write cleared it,
  but resolvers write through store._conn and skip that. A new import
  added that way was never seen: the next pass left the call bare. The
  call pass now always builds it fresh, and the tested-by pass drops it,
  so it never outlives the pair or holds every node name in a watch
  store.
- GraphStore.close() and EmbeddingStore.close() ran gc.collect(). Every
  MCP tool call closes a store, and a full collection costs 10-15 ms in
  the server process, so it moved to the end of a build only.
- EmbeddingStore set temp_store=MEMORY, the opposite of the graph fix.
- Removed the batch writer leftovers: pending_batch was never filled and
  _can_batch_store was never called.
- _single_hop_dependents and detect_communities sniffed whether a test
  had patched a GraphStore method. The dependents query is now
  GraphStore.get_direct_dependent_files, and the community edge scan
  always runs.
- parser.py had its own copy of normalize_file_path; it re-exports the
  one in constants.py.
- Restored the comments the PR deleted in the evidence builder.

Each behaviour has a test in tests/test_derived_data_fallbacks.py that
fails on the PR head.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HengYangDS pushed a commit to HengYangDS/code-review-graph that referenced this pull request Nov 9, 2026
tirth8205#1139)

main() opens the watch store with GraphStore(...), but the PR imported
GraphStore only under TYPE_CHECKING and served it through the module
__getattr__, which a bare global lookup never reaches. The incremental
module (and so graph.py) loads with the server anyway, so GraphStore and
find_project_root are imported normally again; only tool and prompt
modules stay deferred.

The worker also guessed which implementation to load from the tool name.
It now loads exactly the names the tool's lambda reads, and a test fails
if a deferred name is read anywhere the worker does not resolve it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HengYangDS pushed a commit to HengYangDS/code-review-graph that referenced this pull request Nov 9, 2026
tirth8205#1139 skips flows and communities when an update stored no file. tirth8205#964
rebinds C# calls on a project-only change and marks the flows dirty, so the
skip left them stale. A dirty mark now ends the skip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HengYangDS pushed a commit to HengYangDS/code-review-graph that referenced this pull request Nov 9, 2026
tirth8205#1139 loads the tool modules on first use. In test_cli.py an update test
patches GraphStore and get_db_path while they load for the first time, so
the mocks stayed bound in them and the real search test from tirth8205#823 that ran
next found nothing. The test conftest now loads them up front.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HengYangDS pushed a commit to HengYangDS/code-review-graph that referenced this pull request Nov 9, 2026
tirth8205#1139 makes graph.py load the parser on first use, so EdgeInfo is only a
type name there. tirth8205#715 builds EdgeInfo objects when it aligns edges with
colliding names, which raised NameError on every store. The two methods
now import it themselves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HengYangDS pushed a commit to HengYangDS/code-review-graph that referenced this pull request Nov 9, 2026
tirth8205#1139 loads igraph on first use through _load_igraph, so setting
IGRAPH_AVAILABLE alone no longer makes detect_communities take the Leiden
path when igraph is not installed. The tirth8205#882 test now stubs the loader too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants