Skip to content

fix(api): stop a big graph read from freezing the event loop (#5703) - #5782

Merged
nicoloboschi merged 3 commits into
mainfrom
fix/5703-graph-event-loop
Oct 19, 2026
Merged

nicoloboschi merged 3 commits into
mainfrom
fix/5703-graph-event-loop

Conversation

@nicoloboschi

@nicoloboschi nicoloboschi commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #5703.

Problem

On a bank with a few thousand observations, the Memories → Constellation view (and "Load more") made GET /banks/{id}/graph freeze the event loop for seconds. /health/live could not answer, and Kubernetes restarted the pod, failing recall/retain for every client.

Only the stored links were capped (10k). The edges the engine derives on top of them (observations inheriting their source memories' links, observations sharing a source, memories sharing an entity) had no cap. A dense bank derived hundreds of thousands, then validated them against the response model, all on the loop.

Fix

  • Cap the derived edges (GRAPH_MAX_EDGES, now shared by the store and the engine). Stored links (with the ones observations inherit) and derived edges each get a 10k budget, so a well-linked bank still shows entity edges. Derivation stops at the cap, shared source before shared entity (rarest entity first, since a name in every memory says nothing).
  • Build in a worker thread, after the DB connection goes back to the pool.
  • Encode with orjson instead of validating thousands of nodes against GraphDataResponse on the loop. The OpenAPI schema is unchanged; tests/test_graph_assembly.py holds the built dict to the model's shape.

No node limit: the Table view uses the same endpoint, and its "Load more" grows limit instead of paging, so capping nodes would break it.

Behaviour changes

  • Graphs that would have derived more than 10k edges now show at most 10k stored + 10k derived, picked in the order above. Smaller graphs get the same edges as before.

Evidence

Synthetic bank (store mocked, machine heavily loaded, so times are inflated):

nodes before after
1,000 205k edges, 61 MB, 72s loop stall 10k edges, 4.3 MB
7,000 did not finish 10k edges, 13 MB, encoding ~20 ms (validation was ~10 s)

23 graph-related tests pass (new test_graph_assembly.py, plus graph filtering, tag scope, row models, cancellation, HTTP integration).

GET /banks/{id}/graph with a few thousand observations froze the API's event
loop for seconds at a time, long enough for the liveness probe to restart the pod.

- Cap the whole answer at GRAPH_MAX_EDGES (10k). Only the stored links were
  capped; the edges derived on top of them (inherited from source memories,
  shared source, shared entity) were not, and a dense bank derived hundreds of
  thousands. Derivation now stops at the cap, keeping stored links first, then
  inherited, shared-source, and shared-entity edges (rarest entity first).
- Build the answer in a worker thread, after the DB connection is released.
- Encode it with orjson instead of validating it against GraphDataResponse on
  the loop; a test holds the built dict to the model's shape.

Synthetic bank, 1,000 nodes: 205k edges / 61 MB / 72s loop stall before;
10k edges / 4.3 MB after. 7,000 nodes: 13 MB, encoding ~20 ms (was ~10 s).
Review fix: with one shared cap, a well-linked bank filled it with stored
links and lost every entity and shared-source edge. Stored (with inherited)
and derived edges now get GRAPH_MAX_EDGES each; test covers the full-budget case.
@nicoloboschi
nicoloboschi merged commit 856b3fc into main Oct 19, 2026
117 checks passed
@nicoloboschi
nicoloboschi deleted the fix/5703-graph-event-loop branch October 19, 2026 13:32
@nicoloboschi nicoloboschi mentioned this pull request Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GET /banks/{bank_id}/graph with a large limit blocks the API event loop long enough for liveness probes to restart the pod

1 participant