Skip to content

Verify: whole-scope reloads cost the Operator latency #469

Closed

Description

@possiblyneal

Part of #448

Question

Do observe/compress reload the whole session per Observation, and do count/list endpoints load full scopes? Measure on a 20k store and against /livez and burst latency; tie to the residual of rohitg00#499. Upstream: rohitg00#1734, rohitg00#1735.

Activity

  1. self-assigned this
    on Nov 18, 2026
  2. possiblyneal commented on Nov 18, 2026

    @possiblyneal
    OwnerAuthor

    #469 verdict (verified at fdc3d45; worktree tmp/wt/verify-469, branch verify/469-scope-reloads)

    Verdict: APPLY rohitg00#1734 in part (the one site present here, evictOverCap), measured: on Sessions under the cap, observe p50 255 → 24 ms and /livez max 328 → 51 ms (median of 5 interleaved rounds). REJECT rohitg00#1735 for latency (no hot-path full-scope list). Bar: typecheck 0 errors; npm test 271 files, 3,117 passed, 1 skipped; eval:gate and eval:gate:rules pass (no metric below floor; Injection p95 under 500 ms). Not class 2 for Recall content (no change to what is stored, evicted or returned), so eval:suite was not run. Docs: no CLAUDE.md text describes the per-Observation list, so none changed; StateKV.count is a new internal method.

    Measured in-process on tmp/verify/live-copy.sqlite through SqliteState, median of 15, host load avg ~26 (so absolute ms are inflated, ratios are not). node:sqlite is synchronous, so every ms below is event-loop time (/livez, hooks).

    Store: 1,551 Session scopes; largest mem:obs: 2,001 / 2,000 / 1,999 / 1,991 / 1,940 / 1,913 rows (3.1-3.6 MB each; MAX_OBS_PER_SESSION default 2000). mem:semantic 40,081 rows (49 MB), mem:audit 52k, mem:sessions 1,538, mem:summaries 1,801 (3 MB), mem:injections 3,533, mem:memories 42.

    rohitg00#1734 (observe/compress reload whole Session per Observation): PARTLY APPLIES, 1 of 3 sites

    • Site 1 (observe dedup list + find under lock): NOT PRESENT here. Observe takes only kv.get of the Session row inside obs lock (src/functions/observe.ts:471-480).
    • Sites 2/3 (compress.ts list + budgetLiveObservationSource): NOT PRESENT. src/functions/observation-source-budget.ts does not exist; compress.ts and compress-batch.ts use only kv.get/kv.set (compress.ts:107,120,199; compress-batch.ts:77). compress-batch.ts:193 lists KV.compressPending only in mem::compress-resume (boot).
    • APPLIES, a different site: evictOverCap (observe.ts:83-98, called at observe.ts:571 for every Observation whenever cap > 0, i.e. by default) does kv.list(KV.observations(session)) after every stored row, even when the Session is far under the cap, then returns 0 if no excess. Cost on the Operator's largest Sessions: 25-37 ms per Observation (parse of 2,000 JSON rows), synchronous, inside the obs: keyed lock. Sort/filter of victims is 1.4 ms; get of one row 0.04 ms; COUNT(*) 0.13 ms.
    • Operator reach: every tool call of every Session (a 32-Session burst = 32 x ~25 ms on the loop per 3 s round, while prompt-context is already 215 ms). Largest Sessions are 3 of the top 6 at or over cap, where the list is still needed to pick victims.
    • Prototype (uncommitted in worktree): StateKV.count -> state::count (SqliteState.count, COUNT(*)), evictOverCap returns 0 when count <= cap. Removes the 25-37 ms for every under-cap Observation (cost -> 0.13 ms). Does NOT help a Session at the cap (list + victim sort every Observation remains); a follow-up would pick victims in SQL (importance/timestamp ORDER BY ... LIMIT excess) instead of listing.
    • Test: test/observe-session-cap.test.ts "does not list the Session's rows while it is under the cap" fails without the fix (2 lists vs 1), passes with it; 11 observe-registering test files pass (94 tests); typecheck 0 errors.
    • A/B (tmp/verify/469/observe-ab.ts, builds copied aside as dist-main/dist-fix): eval:burst cannot show this, since its burst Sessions hold ~10 Observations each and never list a large scope. Instead: a throwaway daemon seeded with 4 Sessions x 1,800 Observations (under the 2,000 cap), then 4 concurrent streams x 50 observe posts into those Sessions while /livez is polled every 50 ms; 5 rounds, arms alternating order each round, fresh copy of the seed per arm-round, load recorded before each.
    Round Load main / fix observe p50 main / fix (ms) observe p95 main / fix /livez max main / fix
    1 25.4 / 24.3 240 / 24 391 / 60 329 / 51
    2 25.1 / 23.3 255 / 54 391 / 138 309 / 136
    3 28.5 / 27.1 277 / 27 401 / 64 328 / 68
    4 26.5 / 26.7 364 / 17 609 / 40 541 / 43
    5 28.0 / 26.8 250 / 15 423 / 35 257 / 46
    Median 255 / 24 401 / 60 328 / 51

    Every round favours the fix, at matched load (host load 23-28 from the concurrent verify chains, so absolute ms are inflated). The at-cap case is unchanged by construction (count > cap falls through to the same list) and was not timed.

    rohitg00#1735 (count/list endpoints load full scopes): REJECT for latency, mostly

    • /agentmemory/health (api.ts:341): one kv.get of mem:health/latest, 0.06 ms. /livez does not touch the store. Not affected.
    • api::sessions -> listSessionsPage (sessions-page.ts:32-39): full list of mem:sessions then slice: 11.5 ms at 1,538 rows. Under 20 ms; viewer-polled only.
    • api::memories ?count=true (api.ts ~2278 path): lists mem:memories, 42 rows, 0.4 ms.
    • agentmemory://status (mcp/server.ts:1204-1206): sessions + memories + health lists, ~12 ms, on demand.
    • src/state/viewer-counts.ts does not exist here; createConsolidationStatusReader pattern not present.
    • Heavier but off the hot path: mem::diagnose categories (diagnostics.ts:159-760; injection-use at :594 lists whole observation scopes per injected Session; sessionless scan :96), api::graph-build (api.ts:1904), replay/sessions (api.ts:672), boot index fill (search.ts:529). On-demand/boot only; semantic 757 ms, insights 120 ms, injections 45 ms, summaries 30 ms per full list.
    • Retry if: the viewer or a hook starts polling mem:semantic, mem:insights, mem:summaries or mem:injections lists, or mem:sessions passes ~5,000 rows (list > 35 ms).

    Residual of rohitg00#499 (unresponsive with concurrent Sessions)

    Not reproducible as a scope reload in compress/health. The per-Observation list in evictOverCap above is the one in-tree contributor to loop hold (sync, per Observation); the rest of rohitg00#499's residual (prompt-context near-twin cap, queueing) was handled by #400 (docs/benchmarks/2026-11-13-session-burst.md). Retry if: eval:burst shows /livez max above ~1 s with the evictOverCap fix in and prompt-context excluded.

    Not measured

    The at-cap Session (count > cap still lists and sorts every Observation). Seeded 20k-row single Session (scales linearly: ~19 ms per 1,000 rows, so ~370 ms) - but MAX_OBS_PER_SESSION caps live Sessions at ~2,000, so it is not a realistic case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions