Skip to content

fix(summarize): token-budget chunking, Session Summary reuse, noop predicate, per-operation breaker - #45

Merged
possiblyneal merged 10 commits into
mainfrom
fix/summarize
Sep 22, 2026
Merged

possiblyneal merged 10 commits into
mainfrom
fix/summarize

Conversation

@possiblyneal

@possiblyneal possiblyneal commented Sep 22, 2026 •

Copy link
Copy Markdown
Owner

Summary

Summarize cluster of #31. Four tracer bullets on top of fix/llm-timeout-hard-cap:

Base commit 01bd9e3 carries the timeout hard-cap removal; those commits are shared with that branch.

Test plan

Closes #31 partially. Refs rohitg00#1367, rohitg00#572, rohitg00#1350.

🤖 Generated with Claude Code

neal added 10 commits September 21, 2026 15:54
fetchWithTimeout clamped every budget to a 170s HARD_BUDGET_CAP_MS while
openai.ts still reported the unclamped value in its timeout error, so an
AGENTMEMORY_LLM_TIMEOUT_MS of 480000 aborted at 170s and claimed 480000ms.
Broker-side records confirm it: 86 client-disconnect 499s on the general
route, p50 duration 169,996ms, p90 170,004ms, none past 400s.

The cap existed to stay under iii's 180s invocation timeout. ADR 0001
deleted that runtime — the in-process Engine has no invocation timeout, so
the ceiling bounded against a limit that no longer exists and turned slow
local models into provider failures, which then tripped the circuit breaker
and failed unrelated calls.

The caller's timeout is now the total budget for all attempts plus sleeps,
honored exactly. test/fetch-timeout.test.ts asserted the clamp; it now
asserts the opposite.

Generated-By: Claude Opus 5 (1M context)
groupKey joined its two parts with a raw NUL byte. One NUL is enough for
git and grep to classify the whole file as binary, so every search over
src/engine/inproc/sdk.ts silently returned nothing and `file` reported it
as data rather than source.

U+001F separates just as well. The key is only ever a streamItems Map key
and is never persisted, so nothing migrates.

Generated-By: Claude Opus 5 (1M context)
CLAUDE.md opened straight into architecture, so every feature argument
started from mechanism. Lead with the goal instead: right Memory, right
moment, right scope, least record that restates it.

Generated-By: Claude Opus 5 (1M context)
Score every open upstream issue on severity x difficulty, cross-link the PRs
that claim to fix each one, and score the 168 PRs that reference no open issue
by how much of their diff lands in files this fork tracks.

Every row carries a disposition: wont-fix for what is out of scope here (no iii
engine, no npm publish, no Windows CI, no deploy tree, no adapter for the host),
already-fixed for what this tree already carries, candidate for the working set.

The 17 high-severity/low-difficulty issues and the 35 high-relevance candidate
PRs were then opened against this tree and read. Rows checked that way carry a
status whose leading token says what the read found; 15 claims turned out to be
already fixed here and moved out of the candidate set. A row with no status is
still upstream's claim, not a verified defect.

Generated-By: Claude Opus 5 (1M context)
Generated-By: Claude Fable 5.1
Every provider is wrapped as resilient(<inner>), so the === "noop" check
never matched and a keyless install ran the whole summarize pipeline.
One predicate looks through the wrapper; summarize and graph Extraction
both use it.

Closes #32. Refs #31, rohitg00#1367.

Generated-By: Claude Fable 5.1
Chunks were cut at a fixed 400 Observations, but per-Observation size
varies 9x across the store, so a capped Session could exceed the broker's
practical latency and time out. Observations are now measured with the
provider's tokenize route (llama.cpp-style POST /tokenize, forwarded
outside the circuit breaker) and estimated at chars/3 when there is none,
then packed in order to SUMMARIZE_CHUNK_TOKENS (default 50,000) with the
system prompt's overhead reserved. SUMMARIZE_CHUNK_SIZE is gone. The
chunk concurrency default drops from 6 to 2: it was tuned for iii's 180s
cap, and the timeout basis is measured at 2 on one GPU.

Closes #33. Refs #31.

Generated-By: Claude Fable 5.1
…g it

A Session Summary is current until an Observation arrives after it was
written; Observations are append-only within a Session, so a stored
summary whose Observation count matches the Session's is returned without
an LLM call. force: true on mem::summarize and POST /agentmemory/summarize
bypasses the reuse. This closes the retry loop on Sessions that summarize
and are stopped again with nothing new.

Closes #34. Refs #31, rohitg00#572.

Generated-By: Claude Fable 5.1
A Session whose summarize calls fail three times used to open the single
breaker and stop compress and extraction for the recovery window, so one
oversized Session stalled every Observation behind it. The resilient
wrapper now keeps a breaker per operation (compress, summarize,
describeImage). circuitState reports the most severe operation so
/agentmemory/health and the CLI keep their shape; circuitStates exposes
the per-operation map and health returns it as circuitBreakers.
describeImage is now forwarded through the wrapper, which it never was.

Closes #35. Refs #31, rohitg00#1350.

Generated-By: Claude Fable 5.1
- Restore CRLF line endings in src/triggers/api.ts; the earlier edit had
  rewritten the whole file to LF.
- Session Summary reuse now keys on Observation count AND the last
  Observation id (new optional SessionSummary.lastObservationId); a
  delete followed by a new Observation no longer passes as current, and a
  summary stored before the field existed is regenerated.
- Always measure tokens when the provider can count; the estimate is only
  the fallback. OpenAIProvider remembers a 404 from /tokenize and stops
  asking for the life of the instance.
- Measure once per summarize, outside the 2-attempt retry, via planChunks.
- Charge each Observation its prompt separator (4 tokens) and floor
  SUMMARIZE_CHUNK_TOKENS at the prompt overhead plus one.
- Chunk fan-out goes through mapWithConcurrency; chunk offsets use a for
  loop.
- isNoopProvider unwraps only balanced resilient(...) layers.

Refs #31

Generated-By: Claude Fable 5.1
@possiblyneal

Copy link
Copy Markdown
Owner Author

Code review: standard scan (Standards, Spec, Correctness/Logic)

All three axes signed off on this head after corrections. Residual notes are non-blocking observations, recorded here so they are not lost:

  • planChunks no longer has the small-Session fast path (every Session is tokenized before chunking).
  • mapWithConcurrency does not pass the item index to its callback.
  • The broker base URL is not stripped of a trailing slash before /tokenize is appended.
  • FallbackChainProvider does not forward describeImage.
  • The Session Summary reuse path returns before metricsStore.record, so a reused summary is not counted.

Reviewed-Head: cbfed9e

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Repair round: token-based summarize chunking, viewer/scrub/context hardening, hook env, API bounds

1 participant