Repository navigation
Description
Summary
Once a state scope grows past 16 MiB, any HTTP endpoint that returns the whole scope (e.g. GET /agentmemory/semantic) makes the worker drop its connection to iii-engine. The worker then reconnects, and every endpoint returns 404/500 until it is back. With a dashboard polling, this becomes a reconnect loop: we measured 280 reconnects in 10 minutes.
The usual reading is "the worker↔engine frame is capped at 16 MiB and not configurable", which leads to purging big scopes. We measured both directions, and the cap only applies to messages the worker sends to the engine. Messages the engine sends to the worker go through up to ~100 MiB. Internal kv.list() calls are therefore fine. What breaks is an endpoint response that embeds a whole scope.
Two unbounded provenance lists make scopes reach that size quickly. Details and proposed fixes below.
Environment
@agentmemory/agentmemory0.9.29 (global install, server =dist/cli.mjs→dist/src-<hash>.mjs)iiiengine 0.11.2,iii-sdk0.11.2 (bundled), state adapterkv/file_based- Node 22, Linux x86_64
Store sizes at the time: mem:insights 159 MB, mem:semantic 34.8 MB, mem:audit 29.3 MB.
Reproduction (both directions, against a live engine, using a throwaway scope)
// repro.mjs — run with: node repro.mjs <mode>
import { registerWorker } from "<agentmemory>/node_modules/iii-sdk/dist/index.mjs";
const mode = process.argv[2];
const sdk = registerWorker("ws://127.0.0.1:49134", { workerName: "repro-" + mode, invocationTimeoutMs: 60000 });
const t0 = Date.now();
const done = (m) => { console.log(mode, m, Date.now() - t0 + "ms"); process.exit(0); };
try {
if (mode === "list") { // engine -> worker
const r = await sdk.trigger({ function_id: "state::list", payload: { scope: "mem:audit" } });
done("OK bytes=" + JSON.stringify(r).length);
} else { // worker -> engine, e.g. mode "set-17"
const n = Number(mode.split("-")[1]) * 1024 * 1024;
await sdk.trigger({ function_id: "state::set", payload: { scope: "repro:frame", key: "k", value: { x: "a".repeat(n) } } });
await sdk.trigger({ function_id: "state::delete", payload: { scope: "repro:frame", key: "k" } });
done("OK");
}
} catch (e) { done("ERROR " + (e?.message ?? e)); }Results:
| Direction | Payload | Result |
|---|---|---|
engine → worker, state::list of mem:audit |
27.7 MB (66,381 items) | OK, 0.9 s |
engine → worker, state::list of a test scope |
84 MiB | OK, 0.4 s |
engine → worker, state::list of a test scope |
112 MiB | fails: Invocation timeout after 90000ms |
worker → engine, state::set |
15 MiB | OK |
worker → engine, state::set |
17 MiB | fails: connection dropped silently, Invocation timeout after 60000ms |
The running agentmemory worker was not affected by these tests. Only the throwaway worker's own connection dropped.
Root cause
- worker → engine: the engine serves the worker WebSocket through axum (
engine/src/workers/worker/ws_handler.rs). This looks like tungstenite's defaultmax_frame_sizeof 16 MiB. The Nodewsclient sends each message as a single frame, so any message over 16 MiB is rejected, the socket closes, and the pending invocation times out. Nothing iniii-config.yamlexposes this setting. - engine → worker: the Node
wsclient uses its defaultmaxPayloadof 100 MiB (iii-sdkdoes not override it). Reads up to that size succeed. - What actually sends > 16 MiB: an HTTP handler such as
api::semantic-listreturns{ semantic: await kv.list(KV.semantic) }. That return value travels worker → engine (to iii-http), so it hits the cap. The same applies to the export endpoint, which returns memories, actions, semantic, procedural, relations and graph together. 130api::*handlers are registered, and several return whole scopes.
Why scopes get that big: two unbounded provenance lists
semantic.sourceSessionIds, inmem::consolidate-pipeline:sourceSessionIds: recentSummaries.map((s) => s.sessionId),
recentSummariesis the latest batch of up to 20 session summaries. The same 20 ids are copied into every new fact of that pass, so this is not real per-fact provenance. In our store this field was 69 % ofmem:semantic's bytes.insights.sourceMemoryIds, inmem::reflect:This holds every semantic fact sharing a term with the cluster and is never capped (same forsourceMemoryIds: cluster.factIds,
sourceLessonIds/sourceCrystalIds).mem:insightsreached 159 MB.
Also,insight-decay-sweeponly rewrites insights whose confidence changed. Insights at the floor, or soft-deleted, are never rewritten, so bounding at write time alone would not shrink the existing scope.
As far as we can tell, nothing in the 0.9.29 bundle reads these fields back, except export/import and mesh sync, which copy them as-is. When we checked the 0.9.30 tarball, both assignments were unchanged. boundSources (slice(-32)) in 0.9.30 only applies to graph sourceObservationIds.
What we deployed locally (works, measured)
1. Paginate api::* responses above a byte budget. We wrap sdk.registerFunction once, right after registerWorker:
{
const budget = Number(process.env.AGENTMEMORY_REPONSE_MAX_OCTETS) || 12 * 1024 * 1024;
const size = (v) => Buffer.byteLength(JSON.stringify(v) ?? "");
const int = (v) => { const n = Number.parseInt(v, 10); return Number.isFinite(n) && n >= 0 ? n : null; };
const paginate = (req, res) => {
if (!res?.body || typeof res.body !== "object" || Array.isArray(res.body)) return res;
const q = req?.query_params || {};
const asked = int(q.offset) !== null || int(q.limit) !== null;
const total = size(res.body);
if (!asked && total <= budget) return res;
const arrays = Object.keys(res.body).filter((k) => Array.isArray(res.body[k]))
.map((k) => [k, size(res.body[k])]).sort((a, b) => b[1] - a[1]);
if (!arrays.length) return res;
const offset = int(q.offset) ?? 0;
const [bigName, bigBytes] = arrays[0];
let limit = int(q.limit) || Math.max(1, Math.floor((budget * 0.9 - (total - bigBytes)) / (bigBytes / (res.body[bigName].length || 1))));
const orig = Object.fromEntries(arrays.map(([k]) => [k, res.body[k]]));
let fields;
for (;;) {
fields = [];
for (const [k] of arrays) {
res.body[k] = orig[k];
if (!asked && size(res.body) <= budget) break;
res.body[k] = orig[k].slice(offset, offset + limit);
fields.push(k);
}
if (size(res.body) <= budget || limit === 1) break;
limit = Math.max(1, Math.floor(limit / 2));
}
const totals = Object.fromEntries(fields.map((k) => [k, orig[k].length]));
const more = fields.some((k) => offset + limit < orig[k].length);
res.body.pagination = { fields, offset, limit, totals, next_offset: more ? offset + limit : null };
return res;
};
const register = sdk.registerFunction.bind(sdk);
sdk.registerFunction = (id, handler, ...rest) => {
if (typeof id !== "string" || !id.startsWith("api::") || typeof handler !== "function") return register(id, handler, ...rest);
return register(id, async (req, ...a) => {
const res = await handler(req, ...a);
try { return paginate(req, res); } catch { return res; }
}, ...rest);
};
}- The response shape is unchanged below the budget.
- Above it, the largest arrays are paged with
?offset=&limit=and the body carriespagination.next_offset. - A client that ignores pagination gets the first page instead of taking the worker down.
Measured in production: GET /agentmemory/semantic (15.7 MB) is served as 2 pages (20,861 + 8,087 facts). All 28,948 facts were rebuilt with no duplicates, in 1.8 s, with 0 worker reconnects. The effective per-scope ceiling moves from 16 MiB to ~100 MiB, the engine → worker limit.
2. Bound the provenance lists at the single state::set choke point (StateKV.set):
mem:semantic: keep the 3 most recentsourceSessionIdsand addsourceSessionCountwith the original length.mem:insights: keep the last 32 ofsourceMemoryIds/sourceLessonIds/sourceCrystalIds.
The decay tier of consolidate-pipeline rewrites every semantic fact on each pass. The existing scope therefore shrank by itself within one pass: 34.8 MB → 16.5 MB, 28,956 facts rewritten, 0 errors. mem:insights had to be purged once (service stopped, file backed up). It regenerated to 100 KB.
Suggested upstream fixes
- Paginate list endpoints (
/agentmemory/semantic,/memories, export, …): addoffset/limit(or a cursor) and returnnext_offset, at least whenever the serialized body would exceed a budget below the frame cap. The viewer would need to follownext_offset. - Do not copy the batch's session ids into every semantic fact. Either record the sessions a fact actually came from, or store a count. If the list stays, cap it the way
boundSourcescaps graph sources. - Cap
insights.source*Ids(slice(-32), likeboundSources). Also give soft-deleted and floor-confidence insights a hard-delete or compaction path; today nothing ever rewrites or removes them. - Engine/SDK side, optional: expose the WebSocket
max_frame_size/max_message_sizein the iii config, or have the SDK fragment large messages. Tungstenite reassembles fragmented messages up tomax_message_size, which defaults to 64 MiB. Either way, failing loudly instead of a silent drop and a 60 s timeout would have saved us a lot of diagnosis.
Happy to open a PR for (1)–(3) against the TypeScript sources if that direction suits you.
Activity
onelaview commentedon Oct 18, 2026
Same failure here on 0.9.30 (macOS, global install via Volta, default
file_basedstate store), so it reproduces after the 0.9.30 auth change too.Store sizes (on-disk
.binfiles under~/.agentmemory/data/state_store.db/):mem:insightsmem:graph:edgesmem:semanticmem:graph:nodesObserved:
GET /agentmemory/semanticreturns500 {"error":{"code":"invocation_stopped",...}}in ~0.4 s. Right after it, other routes return 404 for ~1 s (e.g./agentmemory/consolidation/status: 404, 404, then 200), and the worker PID doesn't change. That matches the drop-and-reconnect you describe.Possibly related: viewer snapshot times out on a cold cache
GET /agentmemory/viewer/snapshotthrough the viewer (:3113) returns504 {"error":"upstream timeout"}.proxyToRestApiaborts the upstream fetch after a hardcoded 10 s.:3111with the bearer token succeeds, but takes 17.7 s cold. The response is only ~19 KB. A repeat call within 30 s takes 0.06 s, so the slow part looks like the health/counts scan inhealthPayload(cached forUNINDEXED_SCAN_REUSE_MS= 30 s), not the payload size.I haven't confirmed that the slow scan is driven by the insights scope size. Given 691 MB in
mem:insights, it seems likely. A configurable (or longer) viewer proxy timeout, or a precomputed count, would avoid the 504 regardless.+1 to fixes (1)–(3), especially capping
insights.source*Ids.Fguarien commentedon Nov 1, 2026
Follow-up after reading the 0.9.31 release:
api::semantic-listinsrc/triggers/api.ts(v0.9.31 and currentmain) still returnskv.list(KV.semantic)whole, with nolimit/cursorand nocheckPayloadFrameSizecall./exportgot that guard in 0.9.30 (#1422) and paginated collections in 0.9.31 (#1143). On this installmem:semanticis still around 35 MB, so the endpoint stays blocked at the reverse proxy.#1447 (page
/semanticand/procedural) looks like the fix for the endpoint half of this issue, and it is still open. Until it merges, wrapping the existing response incheckPayloadFrameSizewould turn the worker drop into a 413, the same behaviour/exporthas had since 0.9.30.fix(sessions): read summaries by key for returned sessions only
Fguarien commentedon Nov 1, 2026
Related whole-scope read, in case it helps scope a fix for this issue:
createConsolidationStatusReader(src/triggers/api.ts, around L471-479 onmain@ c6c476d) runskv.list()on five entire scopes —summaries,memories,semantic,procedural,relations— only to compute counts (summaries.length,semantic.length, …, plus one filter over memories). On a large store this is the same failure mode as/semantichere: a singlestate::listframe can exceed the engine's 16 MiB limit and drop the worker, after which unrelated endpoints fail until it reconnects.A count primitive (or paginated counting) in
StateKVwould avoid materialising these scopes just to size them. ThesingleFlight+ reuse window limits how often it happens, but not the frame size of each call.(Observed while reviewing #1649, which was closed after the reviewers pointed out the same frame-size risk in a bulk summaries read.)
rohitg00 commentedon Nov 9, 2026
Fixed in 0.9.33. The frame limit is 64 MiB (#1634). Whole-scope reads page, and unpaged list endpoints that would exceed the limit return 413 with a paging hint instead of dropping the connection (#1636).