Skip to content

feat(runtime): bind routes and state reactions through iii 0.25 observers - #1637

Open
rohitg00 wants to merge 16 commits into
mainfrom
feat/iii025-s1b-observe-bindings
Open

rohitg00 wants to merge 16 commits into
mainfrom
feat/iii025-s1b-observe-bindings

Conversation

@rohitg00

@rohitg00 rohitg00 commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

Summary

Stage 1b of the iii 0.25 stack. It targets #1636, which targets #1634.

  • One bindings layer (src/runtime/bindings.ts, getBindings(sdk)) for all 152 HTTP routes and 12 state reactions:
    • When the http/state workers have the new observable functions (http::on-request in http 0.22.8, state::on-changed in state 0.22.22, both pinned by feat(runtime): iii 0.25-built state/http workers and a 64 MiB transport cap #1634), routes and reactions become observers. agentmemory then makes no deprecated registerTrigger calls, and the deprecation warning is gone.
    • Mode is detected per provider via engine::functions::list, waiting up to 30 s. If a provider lacks the function, it falls back to registerTrigger.
    • A route or reaction is never bound both ways. A route whose config has fields http::on-request can't express falls back to a trigger on its own.
    • Kill switch: AGENTMEMORY_BINDINGS=trigger. Test SDKs without observe also take the trigger path, with an identical call object, so existing tests keep passing.
    • State observers use deliver: "input", so handlers get the same payload as before. Graph scopes add a :generation: prefix filter, and key-only handlers (latest, consolidation:lastPipelineRun, current) add a key filter on top of their existing in-handler check. Viewer streams run one event at a time (max_in_flight: 1) to keep order.
  • Boot readiness and re-arm (Worker Compose boot race: first state reads fail with state::get not found, and vector recovery stays paused until restart #1623, Deploy stack: startup index backfill fails silently (function_not_found at boot is terminal — smart-search empty until manual restart) #1631):
    • Before the first boot read there is a bounded, non-fatal state::get probe (AGENTMEMORY_STATE_READY_TIMEOUT_MS, default 60 s).
    • Index recovery and capture recovery are now idempotent, and run again when the state worker comes back. In observe mode, "back" means state observers returning from pending to active, or an engine::on-workers-changed observer followed by a probe. In trigger mode, a 30 s probe tick does it.
  • Duplicate-worker detection: engine::workers::list is checked at boot and on every re-arm. Duplicates show up in /agentmemory/status under runtime.duplicateWorkers, as a viewer alert, and as the duplicate-workers doctor check.
  • Status gains runtime.bindings (modes, observer count, reasons) and runtime.stateWorker.

New source files are in src/runtime/ (bindings.ts, state-worker.ts, recovery.ts, duplicates.ts). Docs touched: AGENTS.md pattern, docs/upgrades/iii-0.25.md, .env.example, compose env lists, CHANGELOG.

Live verification (engine 0.25.0, state 0.22.22, http 0.22.8, Worker Compose)

check auto (observe) AGENTMEMORY_BINDINGS=trigger
runtime.bindings observe / observe, 164 observers trigger / trigger
engine::observers::list 165, all active 0
livez / remember / search 200 / 201 / 200 200 / 201 / 200
viewer SSE memory.created after a remember yes yes
registerTrigger is deprecated warning absent present (expected)
kill state worker → compose::up re-arm ran, search 200 afterwards n/a
kill http worker 90 s → compose::up 152 route observers pending (OWNER_UNAVAILABLE), then active within 5 s, REST 200 routes back within 5 s, REST 200

The live counts (165 observed, 164 reported) don't match the planned 164 (152 routes + 12 state reactions) plus the workers-changed observer; this needs a recount.

  • Cold-boot loop: startup.mjs --repeat 5, all 5 runs passed 17/17.
  • gate:large-store --small --assert: pass.

Test plan

  • npm run build, npm run skills:gen, npm run docs:sync / docs:check, npm run skills:check
  • npm test: 4125 passed, 6 skipped
  • New test/bindings.test.ts (10 cases): observe and trigger modes, env override, SDK without observe, graph and key filters, route ids, unsupported-config fallback, idempotent activate, specs registered during detection, ready() rejection logging, singleton
  • Live checks in the table above

Fixes #1623. Fixes #1631. Refs #1626 #1627 #1603 #1593.

Summary by CodeRabbit

  • New Features
    • Runtime bindings can select observer or trigger mode automatically, with options to set readiness timeouts and delivery limits.
    • Status reports and the dashboard display runtime health and warn about duplicate workers; diagnostics can also detect duplicates.
    • State-worker readiness and recovery support resuming index and capture processing when a worker returns.
  • Bug Fixes
    • Status checks flag failed observers, and recovery can retry unavailable checkpoints while preserving eligible index data.
    • Oversized export errors point to project-scoped exports.
  • Documentation
    • Updated setup, configuration, and upgrade guidance, and refreshed the documented test count.

rohitg00 and others added 5 commits October 29, 2026 15:25
…nsport cap to 64 MiB

state 0.22.20 and http 0.22.7 are built on iii-sdk 0.24.6, which caps outgoing
messages at 16 MiB, so any whole-scope state read past that failed with
payload_too_large. state 0.22.22 and http 0.22.8 are iii-sdk 0.25.0 builds that
fragment messages up to 64 MiB, so the payload guard moves to 60 MiB.

Graph index migration now appends the underlying error to its storage-failure
message, AGENTMEMORY_LOCAL_DISPATCH=false sends internal calls back through the
engine for full traces, and gate:large-store measures large scopes against a
real Compose runtime. The startup gate gains --repeat N for cold-boot loops.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Signed-off-by: Rohit Ghumare <ghumare64@gmail.com>
…errupts in repeat mode

The large-store gate now waits for the agentmemory worker to appear in
engine::workers::list before taking the baseline, and treats an empty or
failed observation as unstable instead of a pass. Repeat mode in the startup
gate forwards SIGINT/SIGTERM/SIGHUP to the active child's process group so
its teardown runs, escalating to SIGKILL on a second signal. The Docker
Compose environments forward AGENTMEMORY_LOCAL_DISPATCH, and the
registration-defaults tests no longer depend on the caller's environment.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Signed-off-by: Rohit Ghumare <ghumare64@gmail.com>
Even with 64 MiB transport messages, a single state::list of the insights or
graph scopes can exceed the limit (524 MB insights, 73 MB graph nodes in the
large-store gate). StateKV gains listKeys, getMany, iterate and listPage built
on state::list_keys plus batched state::get, and switches to state::get_many
when the state worker provides it. Every reader of an unbounded scope now goes
through them, and a guard test blocks new whole-scope reads.

Mesh export pages with a stable id cursor and the sync client follows it to
the end, stamping lastSyncAt with the sync start time only after the final
page. Export no longer swallows collection read failures, which silently
dropped semantic and insight data when the read failed. Graph index migration
discovers ids through listKeys.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Signed-off-by: Rohit Ghumare <ghumare64@gmail.com>
… imports consistent

Mesh export now emits a cursor when a page fills exactly at a collection
boundary, so later collections are no longer dropped. The sync watermark is
the per-peer start time and only advances when every requested direction
succeeded, so a failed push is retried. The speculative state::get_many path
is removed until the state worker offers a byte-bounded batch protocol.
collectAll restores global insertion order on the redis backend. Replace
imports read every row they will delete before deleting anything, so a read
failure no longer leaves a half-deleted store.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Signed-off-by: Rohit Ghumare <ghumare64@gmail.com>
…vers

All 152 HTTP routes and 12 state reactions now go through one bindings layer.
With state 0.22.22 and http 0.22.8 it observes state::on-changed and
http::on-request, so agentmemory no longer uses the deprecated registerTrigger.
It falls back to registerTrigger per provider when the observable function is
missing, never binds a spec both ways, and AGENTMEMORY_BINDINGS=trigger keeps
the old path. State filters cover graph generation scopes and key-only
handlers.

Boot waits for the state worker with a bounded, non-fatal probe
(AGENTMEMORY_STATE_READY_TIMEOUT_MS). Index and capture recovery run again when
the state worker comes back, detected by state observers returning to active or
an engine::on-workers-changed observer, with a probe tick in trigger mode.
Duplicate agentmemory workers are reported in status, the viewer and doctor.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Signed-off-by: Rohit Ghumare <ghumare64@gmail.com>
@vercel

vercel Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
agentmemory Ready Ready Preview Nov 8, 2026 3:48pm UTC

Request Review

@coderabbitai

coderabbitai Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • Resume reviews
  • Trigger review
Walkthrough

The pull request adds runtime bindings that select observer or trigger implementations for HTTP and state handlers. It adds state-worker readiness and recovery paths, runtime and duplicate-worker diagnostics, and updates deployment configuration, documentation, tests, and release checks.

Changes: Runtime bindings and recovery

Layer / File(s) Summary
Binding selection and handler registration
src/runtime/bindings.ts, src/triggers/*, src/mcp/server.ts, src/state/graph-generation.ts, test/bindings.test.ts, src/cli/compose-project.ts, docker-compose.yml, deploy/*, .env.example, AGENTS.md, docs/upgrades/*, plugin/skills/agentmemory-config/*, scripts/skills/rest-endpoints.ts
HTTP and state handlers register through getBindings(sdk). The runtime selects observers or triggers, applies configured limits, and records observer readiness failures. Deployment settings, configuration references, usage guidance, and binding tests are updated. Graph state reads use the pinned generation context.
State-worker readiness and recovery
src/runtime/state-worker.ts, src/runtime/recovery.ts, src/index.ts, src/state/index-persistence.ts, src/state/vector-index.ts, test/state-worker-return.test.ts
Startup probes state-worker readiness and restores checkpoints. Worker-return handling can resume recovery paused by unavailable checkpoints, re-run eligible index recovery, rebuild a stale session index, and sweep unfinished capture work.
Runtime status and duplicate-worker reporting
src/runtime/duplicates.ts, src/functions/status.ts, src/triggers/api.ts, src/cli.ts, src/cli/doctor-diagnostics.ts, src/viewer/index.html, test/duplicate-workers.test.ts, test/viewer-dashboard-reliability.test.ts
Status reports include binding, state-worker, and duplicate-worker data. Status evaluation reports failed observers. The CLI adds a manual duplicate-worker diagnostic, and the dashboard displays a warning when more than one worker is reported.
Release checks and supporting updates
scripts/release-gate/large-store.mjs, scripts/docs-sync.state.json, README.md, READMEs/*, test/mesh-export-project-scope.test.ts, src/triggers/api.ts
The release probe detects a matching success:false response when JSON parsing fails. The oversized-export hint no longer names ?collections. Repository statistics and documented test counts are updated.

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant index.ts
  participant StateWorker
  participant IndexPersistence
  participant CaptureRecovery
  index.ts->>StateWorker: probe state::get during startup
  StateWorker-->>index.ts: return readiness result
  index.ts->>IndexPersistence: restore checkpoint and prepare recovery
  StateWorker-->>index.ts: signal worker return
  index.ts->>IndexPersistence: retry checkpoint and eligible index recovery
  index.ts->>CaptureRecovery: sweep unfinished observations
Loading

Possibly related PRs

  • rohitg00/agentmemory#1592 — This PR updates viewer and event registration paths that were shaped by the Worker Compose migration.
Merge Risk: Moderate · previous review 80001

Checkpoint recovery can retain deleted search candidates, lose newly embedded vectors, or reload stale embeddings after restart. Resolve these recovery interleavings before merging, or explicitly accept their bounded semantic-search impact.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title "feat(runtime): bind routes and state reactions through iii 0.25 observers" matches the main change. The PR adds a shared bindings layer (src/runtime/bindings.ts) that routes HTTP and state …
Linked Issues check Passed Both directly linked issues, #1623 and #1631, have State: closed. Both are closed issues, so they supply historical context only. No active directly linked target remains. No linked-issue coding requi…
Out of Scope Changes check Passed The whole-PR changes from merge base ad14556 to head 800016d map to the PR description. Bindings layer: src/runtime/bindings.ts, with getBindings call sites in src/mcp/server.ts, src/triggers/api.ts, …
Full details: Title check

Explanation

The title "feat(runtime): bind routes and state reactions through iii 0.25 observers" matches the main change. The PR adds a shared bindings layer (src/runtime/bindings.ts) that routes HTTP and state registrations through iii 0.25 observers, with registerTrigger as fallback. The title does not mention the other additions: state-worker readiness probing, index and capture recovery re-runs, duplicate-worker detection, and the doctor and status diagnostics. Those are secondary to the main change, so the title is a fair summary.

Full details: Linked Issues check

Explanation

Both directly linked issues, #1623 and #1631, have State: closed. Both are closed issues, so they supply historical context only. No active directly linked target remains. No linked-issue coding requirements apply. The 'Fixes #1623' and 'Fixes #1631' statements in the PR description do not create active requirements. The #1623 acceptance checks and the #1631 suggested fixes are not assessed as requirements. No parent or sibling issue is linked. The #1623 issue comment from ABrocksdorff is background only. The live observer count mismatch (165 listed, 164 reported) is a verification question, not an issue requirement.

Full details: Out of Scope Changes check

Explanation

The whole-PR changes from merge base ad14556 to head 800016d map to the PR description. Bindings layer: src/runtime/bindings.ts, with getBindings call sites in src/mcp/server.ts, src/triggers/api.ts, events.ts, viewer-sse.ts, and viewer-streams.ts. Boot readiness and re-arm: src/runtime/state-worker.ts, src/runtime/recovery.ts, src/index.ts, src/state/index-persistence.ts (resumeAfterUnavailable), src/state/vector-index.ts (trackRemovals, snapshotLive, restoreLive, mergeMissingFrom), and src/state/graph-generation.ts (GRAPH_SCOPES export and pinned wrappers for graph scope filters). Duplicate-worker detection: src/runtime/duplicates.ts, src/functions/status.ts, src/cli.ts (daemonStatus), src/cli/doctor-diagnostics.ts, and src/viewer/index.html. Configuration and docs: .env.example, AGENTS.md, compose files, the compose-project allowlist, docs/upgrades/iii-0.25.md, the config REFERENCE, CHANGELOG, and README translations. The README and docs-sync count updates follow from the new test count. The head commit 800016d (keep live vectors through a checkpoint retry) and commit 6dc2c58 (re-arm after checkpoint save errors) address the boot re-arm and checkpoint behavior described in the PR. Two changes are not named in the PR description: (1) the mesh export 413 hint in src/triggers/api.ts, which now names ?project and no longer names ?collections. test/mesh-export-project-scope.test.ts asserts this text. The change is small, user-visible, and test-covered, but it is not connected to bindings or boot readiness in the description. It is not shown to be unrelated. (2) The large-store probe change in scripts/release-gate/large-store.mjs, which the PR uses for gate verification. Both have test or verification coverage. The duplicate-workers doctor fix returns an unsuccessful result and does not perform the stop/restart action it describes. This is a design gap, not an out-of-scope change. This assessment relies on the whole-PR summary and two focused inspections (commit history and the mesh export test). It did not inspect the mesh hunk in src/triggers/api.ts.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 15 functions across 24 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches

Generate docstrings

  • Commit to this branch
  • Create a new PR
Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

devin-ai-integration[bot]

This comment was marked as resolved.

Owner-delivered http::on-request observers default to 64 deliveries in flight,
so the 65th concurrent request failed with OVERFLOW where trigger-bound routes
had queued. Route observers now set max_in_flight from
AGENTMEMORY_ROUTE_MAX_IN_FLIGHT (default 1024); a 300-request burst returns 201
for every request. Also drops a stray code fence in AGENTS.md.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Signed-off-by: Rohit Ghumare <ghumare64@gmail.com>
When boot runs before the state worker answers, vector recovery was paused
until restart. The re-arm now reloads the checkpoint after the store comes
back, but only when the boot load failed because the checkpoint was
unavailable. The trigger-mode probe starts from the boot readiness result. A
re-arm that arrives inside the throttle window is retried once the window ends
instead of being dropped. The viewer's duplicate-worker alert reads the
status report it is actually given.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Signed-off-by: Rohit Ghumare <ghumare64@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

@ABrocksdorff

Copy link
Copy Markdown

Real-world confirmation of the race this waits for, posted on #1623: #1623 (comment) — a cold Worker Compose boot here registered state::list about 2.7 s after the agentmemory worker, the 3 x 250 ms window in runKeywordRebuild gave up, and the result was 7440 observations plus every memory missing from the keyword index, with the persisted vector checkpoint refused as well.

waitForStateWorker + AGENTMEMORY_STATE_READY_TIMEOUT_MS (60 s, non-fatal) and the re-arm on worker return cover all three effects we measured, which the narrower keyword-rebuild retry in #1626 does not. We would rather see this land than push our own workaround. +1 from our side.

Base automatically changed from feat/iii025-s1a-paged-reads to main November 8, 2026 07:10
coderabbitai[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

This branch carried an earlier copy of the paged-reads work that main now
holds as #1636. Keep main for those files and apply only the observer
binding commits on top. Index recovery keeps the single-flight wrapper
and main's superseded-revision handling, and the boot wait from main
runs before the observer status probe.
- Retry an unavailable vector checkpoint before index recovery on a state
  worker return, and rebuild once it is restored.
- Merge a retried checkpoint into the live vector index instead of
  replacing it, so vectors embedded after boot stay searchable.
- Count trigger-mode routes, bound each function-list probe by the
  detection deadline, and report rejected observers in status.
- Load the status report from the dashboard so the duplicate-worker
  alert shows without visiting Health.
- Drop the ?collections hint the mesh export never honored, detect a
  truncated success:false body in the large-store gate, and leave
  CHANGELOG to the release PR.
coderabbitai[bot]

This comment was marked as resolved.

… current

- Re-run index recovery on a state worker return only after an incomplete
  keyword rebuild, a paused checkpoint or a failed recovery run. A backfill
  paused at its per-start cap is no longer resumed without opt-in.
- Rebuild a stale session index on a state worker return.
- Keep runtime.stateWorker.ready in step with probe and observer state.
- Refresh the duplicate-worker report on the 30 s tick, and read the
  engine worker id field.
- Send the doctor status request with headers for the daemon base URL.
devin-ai-integration[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

…leted

- Re-run index recovery on a state worker return when the vector
  checkpoint has a save error.
- Remember vectors deleted while absent from the live index, so a
  retried checkpoint merge does not restore them.
- Bind the lesson-get route through Bindings.
- Give the doctor status request 8 s, above the server check budget.
devin-ai-integration[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

- Snapshot the live vector index before a retried checkpoint load and
  re-apply it after pending-log replay, so replay cannot overwrite newer
  embeddings or restore deleted vectors.
- Track removed ids only while a boot checkpoint retry is pending.
- Re-arm recovery after a backfill batch that embedded nothing and had
  failures, which stays within the per-start cap.
devin-ai-integration[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

…g a checkpoint retry

While a boot checkpoint is unavailable, the vector index records every id
added or removed live. Checkpoint merge and pending-log replay leave those
ids alone, including a replayed clear, so replay can no longer overwrite a
newer embedding, bring back a deleted vector or delete a re-added one.
Tracking ends only after a retry restores the checkpoint; at that point
live deletions are queued for the next save and the id set is dropped.
devin-ai-integration[bot]

This comment was marked as resolved.

…eckpoint load

Routes are active before the boot checkpoint read, so a live vector write
during that read could be replaced by the checkpoint or an older pending
log entry. Tracking now starts before the load, the boot restore keeps
live-changed vectors and their pending changes, and tracking ends right
after boot when the checkpoint was available.
…ve-bindings

# Conflicts:
#	scripts/docs-sync.state.json

This branch was successfully deployed

1 active deployment
Preview — 525eba76 Deployed Nov 8, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants