Skip to content

Worker drops its iii connection when an endpoint response exceeds 16 MiB (worker→engine frame cap); semantic/insights provenance lists are unbounded #1579

Closed

Description

@Fguarien

Summary

Once a state scope grows past 16 MiB, any HTTP endpoint that returns the whole scope (e.g. GET /agentmemory/semantic) makes the worker drop its connection to iii-engine. The worker then reconnects, and every endpoint returns 404/500 until it is back. With a dashboard polling, this becomes a reconnect loop: we measured 280 reconnects in 10 minutes.

The usual reading is "the worker↔engine frame is capped at 16 MiB and not configurable", which leads to purging big scopes. We measured both directions, and the cap only applies to messages the worker sends to the engine. Messages the engine sends to the worker go through up to ~100 MiB. Internal kv.list() calls are therefore fine. What breaks is an endpoint response that embeds a whole scope.

Two unbounded provenance lists make scopes reach that size quickly. Details and proposed fixes below.

Environment

  • @agentmemory/agentmemory 0.9.29 (global install, server = dist/cli.mjs → dist/src-<hash>.mjs)
  • iii engine 0.11.2, iii-sdk 0.11.2 (bundled), state adapter kv / file_based
  • Node 22, Linux x86_64

Store sizes at the time: mem:insights 159 MB, mem:semantic 34.8 MB, mem:audit 29.3 MB.

Reproduction (both directions, against a live engine, using a throwaway scope)

// repro.mjs — run with: node repro.mjs <mode>
import { registerWorker } from "<agentmemory>/node_modules/iii-sdk/dist/index.mjs";
const mode = process.argv[2];
const sdk = registerWorker("ws://127.0.0.1:49134", { workerName: "repro-" + mode, invocationTimeoutMs: 60000 });
const t0 = Date.now();
const done = (m) => { console.log(mode, m, Date.now() - t0 + "ms"); process.exit(0); };
try {
  if (mode === "list") {                       // engine -> worker
    const r = await sdk.trigger({ function_id: "state::list", payload: { scope: "mem:audit" } });
    done("OK bytes=" + JSON.stringify(r).length);
  } else {                                     // worker -> engine, e.g. mode "set-17"
    const n = Number(mode.split("-")[1]) * 1024 * 1024;
    await sdk.trigger({ function_id: "state::set", payload: { scope: "repro:frame", key: "k", value: { x: "a".repeat(n) } } });
    await sdk.trigger({ function_id: "state::delete", payload: { scope: "repro:frame", key: "k" } });
    done("OK");
  }
} catch (e) { done("ERROR " + (e?.message ?? e)); }

Results:

Direction Payload Result
engine → worker, state::list of mem:audit 27.7 MB (66,381 items) OK, 0.9 s
engine → worker, state::list of a test scope 84 MiB OK, 0.4 s
engine → worker, state::list of a test scope 112 MiB fails: Invocation timeout after 90000ms
worker → engine, state::set 15 MiB OK
worker → engine, state::set 17 MiB fails: connection dropped silently, Invocation timeout after 60000ms

The running agentmemory worker was not affected by these tests. Only the throwaway worker's own connection dropped.

Root cause

  • worker → engine: the engine serves the worker WebSocket through axum (engine/src/workers/worker/ws_handler.rs). This looks like tungstenite's default max_frame_size of 16 MiB. The Node ws client sends each message as a single frame, so any message over 16 MiB is rejected, the socket closes, and the pending invocation times out. Nothing in iii-config.yaml exposes this setting.
  • engine → worker: the Node ws client uses its default maxPayload of 100 MiB (iii-sdk does not override it). Reads up to that size succeed.
  • What actually sends > 16 MiB: an HTTP handler such as api::semantic-list returns { semantic: await kv.list(KV.semantic) }. That return value travels worker → engine (to iii-http), so it hits the cap. The same applies to the export endpoint, which returns memories, actions, semantic, procedural, relations and graph together. 130 api::* handlers are registered, and several return whole scopes.

Why scopes get that big: two unbounded provenance lists

  1. semantic.sourceSessionIds, in mem::consolidate-pipeline:
    sourceSessionIds: recentSummaries.map((s) => s.sessionId),
    recentSummaries is the latest batch of up to 20 session summaries. The same 20 ids are copied into every new fact of that pass, so this is not real per-fact provenance. In our store this field was 69 % of mem:semantic's bytes.
  2. insights.sourceMemoryIds, in mem::reflect:
    sourceMemoryIds: cluster.factIds,
    This holds every semantic fact sharing a term with the cluster and is never capped (same for sourceLessonIds / sourceCrystalIds). mem:insights reached 159 MB.
    Also, insight-decay-sweep only rewrites insights whose confidence changed. Insights at the floor, or soft-deleted, are never rewritten, so bounding at write time alone would not shrink the existing scope.

As far as we can tell, nothing in the 0.9.29 bundle reads these fields back, except export/import and mesh sync, which copy them as-is. When we checked the 0.9.30 tarball, both assignments were unchanged. boundSources (slice(-32)) in 0.9.30 only applies to graph sourceObservationIds.

What we deployed locally (works, measured)

1. Paginate api::* responses above a byte budget. We wrap sdk.registerFunction once, right after registerWorker:

{
  const budget = Number(process.env.AGENTMEMORY_REPONSE_MAX_OCTETS) || 12 * 1024 * 1024;
  const size = (v) => Buffer.byteLength(JSON.stringify(v) ?? "");
  const int = (v) => { const n = Number.parseInt(v, 10); return Number.isFinite(n) && n >= 0 ? n : null; };
  const paginate = (req, res) => {
    if (!res?.body || typeof res.body !== "object" || Array.isArray(res.body)) return res;
    const q = req?.query_params || {};
    const asked = int(q.offset) !== null || int(q.limit) !== null;
    const total = size(res.body);
    if (!asked && total <= budget) return res;
    const arrays = Object.keys(res.body).filter((k) => Array.isArray(res.body[k]))
      .map((k) => [k, size(res.body[k])]).sort((a, b) => b[1] - a[1]);
    if (!arrays.length) return res;
    const offset = int(q.offset) ?? 0;
    const [bigName, bigBytes] = arrays[0];
    let limit = int(q.limit) || Math.max(1, Math.floor((budget * 0.9 - (total - bigBytes)) / (bigBytes / (res.body[bigName].length || 1))));
    const orig = Object.fromEntries(arrays.map(([k]) => [k, res.body[k]]));
    let fields;
    for (;;) {
      fields = [];
      for (const [k] of arrays) {
        res.body[k] = orig[k];
        if (!asked && size(res.body) <= budget) break;
        res.body[k] = orig[k].slice(offset, offset + limit);
        fields.push(k);
      }
      if (size(res.body) <= budget || limit === 1) break;
      limit = Math.max(1, Math.floor(limit / 2));
    }
    const totals = Object.fromEntries(fields.map((k) => [k, orig[k].length]));
    const more = fields.some((k) => offset + limit < orig[k].length);
    res.body.pagination = { fields, offset, limit, totals, next_offset: more ? offset + limit : null };
    return res;
  };
  const register = sdk.registerFunction.bind(sdk);
  sdk.registerFunction = (id, handler, ...rest) => {
    if (typeof id !== "string" || !id.startsWith("api::") || typeof handler !== "function") return register(id, handler, ...rest);
    return register(id, async (req, ...a) => {
      const res = await handler(req, ...a);
      try { return paginate(req, res); } catch { return res; }
    }, ...rest);
  };
}
  • The response shape is unchanged below the budget.
  • Above it, the largest arrays are paged with ?offset=&limit= and the body carries pagination.next_offset.
  • A client that ignores pagination gets the first page instead of taking the worker down.

Measured in production: GET /agentmemory/semantic (15.7 MB) is served as 2 pages (20,861 + 8,087 facts). All 28,948 facts were rebuilt with no duplicates, in 1.8 s, with 0 worker reconnects. The effective per-scope ceiling moves from 16 MiB to ~100 MiB, the engine → worker limit.

2. Bound the provenance lists at the single state::set choke point (StateKV.set):

  • mem:semantic: keep the 3 most recent sourceSessionIds and add sourceSessionCount with the original length.
  • mem:insights: keep the last 32 of sourceMemoryIds / sourceLessonIds / sourceCrystalIds.

The decay tier of consolidate-pipeline rewrites every semantic fact on each pass. The existing scope therefore shrank by itself within one pass: 34.8 MB → 16.5 MB, 28,956 facts rewritten, 0 errors. mem:insights had to be purged once (service stopped, file backed up). It regenerated to 100 KB.

Suggested upstream fixes

  1. Paginate list endpoints (/agentmemory/semantic, /memories, export, …): add offset/limit (or a cursor) and return next_offset, at least whenever the serialized body would exceed a budget below the frame cap. The viewer would need to follow next_offset.
  2. Do not copy the batch's session ids into every semantic fact. Either record the sessions a fact actually came from, or store a count. If the list stays, cap it the way boundSources caps graph sources.
  3. Cap insights.source*Ids (slice(-32), like boundSources). Also give soft-deleted and floor-confidence insights a hard-delete or compaction path; today nothing ever rewrites or removes them.
  4. Engine/SDK side, optional: expose the WebSocket max_frame_size / max_message_size in the iii config, or have the SDK fragment large messages. Tungstenite reassembles fragmented messages up to max_message_size, which defaults to 64 MiB. Either way, failing loudly instead of a silent drop and a 60 s timeout would have saved us a lot of diagnosis.

Happy to open a PR for (1)–(3) against the TypeScript sources if that direction suits you.

Activity

  1. onelaview commented on Oct 18, 2026

    @onelaview

    Same failure here on 0.9.30 (macOS, global install via Volta, default file_based state store), so it reproduces after the 0.9.30 auth change too.

    Store sizes (on-disk .bin files under ~/.agentmemory/data/state_store.db/):

    Scope Size
    mem:insights 691 MB
    mem:graph:edges 40 MB
    mem:semantic 25 MB (~20.6k facts)
    mem:graph:nodes 18 MB

    Observed:

    • GET /agentmemory/semantic returns 500 {"error":{"code":"invocation_stopped",...}} in ~0.4 s. Right after it, other routes return 404 for ~1 s (e.g. /agentmemory/consolidation/status: 404, 404, then 200), and the worker PID doesn't change. That matches the drop-and-reconnect you describe.

    Possibly related: viewer snapshot times out on a cold cache

    • GET /agentmemory/viewer/snapshot through the viewer (:3113) returns 504 {"error":"upstream timeout"}. proxyToRestApi aborts the upstream fetch after a hardcoded 10 s.
    • The same request directly against :3111 with the bearer token succeeds, but takes 17.7 s cold. The response is only ~19 KB. A repeat call within 30 s takes 0.06 s, so the slow part looks like the health/counts scan in healthPayload (cached for UNINDEXED_SCAN_REUSE_MS = 30 s), not the payload size.
    • The result is that the dashboard fails on first load. Reloading within 30 s usually works.

    I haven't confirmed that the slow scan is driven by the insights scope size. Given 691 MB in mem:insights, it seems likely. A configurable (or longer) viewer proxy timeout, or a precomputed count, would avoid the 504 regardless.

    +1 to fixes (1)–(3), especially capping insights.source*Ids.

  2. Fguarien commented on Nov 1, 2026

    @Fguarien
    Author

    Follow-up after reading the 0.9.31 release: api::semantic-list in src/triggers/api.ts (v0.9.31 and current main) still returns kv.list(KV.semantic) whole, with no limit/cursor and no checkPayloadFrameSize call. /export got that guard in 0.9.30 (#1422) and paginated collections in 0.9.31 (#1143). On this install mem:semantic is still around 35 MB, so the endpoint stays blocked at the reverse proxy.

    #1447 (page /semantic and /procedural) looks like the fix for the endpoint half of this issue, and it is still open. Until it merges, wrapping the existing response in checkPayloadFrameSize would turn the worker drop into a 413, the same behaviour /export has had since 0.9.30.

  3. Fguarien commented on Nov 1, 2026

    @Fguarien
    Author

    Related whole-scope read, in case it helps scope a fix for this issue:

    createConsolidationStatusReader (src/triggers/api.ts, around L471-479 on main @ c6c476d) runs kv.list() on five entire scopes — summaries, memories, semantic, procedural, relations — only to compute counts (summaries.length, semantic.length, …, plus one filter over memories). On a large store this is the same failure mode as /semantic here: a single state::list frame can exceed the engine's 16 MiB limit and drop the worker, after which unrelated endpoints fail until it reconnects.

    A count primitive (or paginated counting) in StateKV would avoid materialising these scopes just to size them. The singleFlight + reuse window limits how often it happens, but not the frame size of each call.

    (Observed while reviewing #1649, which was closed after the reviewers pointed out the same frame-size risk in a bulk summaries read.)

  4. rohitg00 commented on Nov 9, 2026

    @rohitg00
    Owner

    Fixed in 0.9.33. The frame limit is 64 MiB (#1634). Whole-scope reads page, and unpaged list endpoints that would exceed the limit return 413 with a paging hint instead of dropping the connection (#1636).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions