Skip to content

fix(runtime): stop checkpoint retry loops and runtime polling in the iii 0.26 observer stack - #1776

Open
guibeira wants to merge 4 commits into
rohitg00:feat/iii026-observersfrom
guibeira:fix/iii026-observers-review
Open

guibeira wants to merge 4 commits into
rohitg00:feat/iii026-observersfrom
guibeira:fix/iii026-observers-review

Conversation

@guibeira

@guibeira guibeira commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #1753. This PR targets feat/iii026-observers, so it lands in #1753 before #1753 merges.

Each issue below was first reproduced with a test that failed on 6c324d7, the current head of #1753. The fix makes that test pass. For issues 1, 3 and 4, the code lives inside main(), so the tests boot the real src/index.ts main() against an in-memory engine (test/helpers/fake-engine.ts). They capture the intervals that main() installs and run the 30 s runtime tick with the clock moved forward.

Before and after

# Scenario Before (6c324d7) After
1a The boot vector checkpoint cannot be read; 10 ticks (5 min) 10 checkpoint reloads, 10 full BM25 rebuilds 3 reloads (backoff), 0 rebuilds
1b The checkpoint comes back with the wrong embedding dimension; 5 ticks 5 "Refusing to start" errors while the worker keeps running, 5 BM25 rebuilds 1 error, no further retries, 0 rebuilds
2 engine::on-workers-changed observer No filter; created outside Bindings, so it is missing from runtime.bindings and gets no OBSERVER_ID_CONFLICT retry Filtered in the engine, owned by Bindings, retried on conflict
3 5 idle ticks 10 engine::workers::list calls 0 while the workers-changed observer is active; the duplicate check runs on connect and disconnect events and falls back to the tick only while that observer is not active
4 After SIGTERM Runtime tick and compression sweep stay scheduled; a late tick still calls the engine Both cleared, re-arm stopped, 0 calls
5 Inactive observers in status Delete cascade, relation reaction, failure audit and post-remember graph observers are reported as running "as direct calls" Only the compression and post-session steps are listed as direct calls; the others are reported separately
6 .env.example header Lines 1 and 2 joined by the merge in 678c67a Matches main

Changes

1. Checkpoint retry loop (src/index.ts, src/state/index-persistence.ts, src/runtime/recovery.ts)

An unreadable boot checkpoint was retried on every 30 s tick, and every retry rebuilt the whole BM25 index. There were three causes:

  • The retry re-set its own "unavailable" flag, so pending() stayed true.
  • A recovery paused by an unreadable checkpoint counted as a reason to rebuild the keyword index.
  • pauseRecovery() writes its message to lastError, so checkpointSaveFailed was also true without any failed save.

The fixes:

  • Retries from the tick back off from 30 s to 30 min (createRetryBackoff). A real state-worker return still retries at once.
  • An unreadable or unusable checkpoint no longer counts as a paused recovery that needs a rebuild.
  • IndexPersistence.checkpointSaveFailed() is true only after a real save failure.
  • A checkpoint with the wrong dimension that is found on a retry now calls pauseForUnusableCheckpoint() instead of prepareBackfill(null). It is logged once, is not retried, and the message no longer says "Refusing to start". The boot path is unchanged.

2. Workers-changed observer (src/runtime/state-worker.ts, src/runtime/bindings.ts)

The observer is now created through Bindings.observeRuntime(). It filters input.event to worker_connected and worker_disconnected in the engine. It now appears in runtime.bindings, and the OBSERVER_ID_CONFLICT retry covers it.

3. engine::workers::list polling (src/index.ts, src/runtime/bindings.ts)

The duplicate-worker check runs at boot, when the state worker returns, and on each worker connect or disconnect event. A disconnect also clears a stale duplicate warning left after a quick restart. The 30 s tick checks only while the workers-changed observer is not active (Bindings.runtimeActive()), and once more when that observer comes back to cover the events it missed. A rejected observer therefore does not leave the warning stale.

4. Shutdown (src/index.ts)

clearInterval(rearmTick), clearInterval(sweepTimer) and rearm.stop() run at the start of shutdown.

5. Status wording (src/runtime/bindings.ts, src/runtime/pipeline-observers.ts, src/functions/status.ts, docs/upgrades/iii-0.26.md)

Pipeline observers now carry a fallback flag. Only mem::compress and mem::post-session-* have a direct-call path; the callers check observing(). The delete cascade, relation reaction, failure audit and mem::post-remember-graph observers are reported as having no direct-call fallback. The observer-rejected status message and the 0.26 upgrade guide now say the same.

6. .env.example: restores the header from main.

Behavior changes

  • The runtime::on-workers-changed observer is now also registered when AGENTMEMORY_BINDINGS=inline. It is runtime infrastructure, not part of the post-capture pipeline.
  • The duplicate-worker warning follows worker connect and disconnect events. The 30 s tick calls engine::workers::list only while that observer is not active.

Tests

  • test/main-runtime-ticks.test.ts: issues 1a, 1b, 3 and 4, and the duplicate-check fallback while the workers-changed observer is rejected, against the real main().
  • test/main-checkpoint-restore.test.ts: a matching checkpoint that becomes readable again is still restored, with one keyword rebuild and no further reloads.
  • test/state-worker-return.test.ts: workers-changed observer filter, ownership and conflict retry; createRetryBackoff; the pauses a retry cannot fix.
  • test/bindings.test.ts: direct-call reasons for issue 5.

Verification

  • npm test: 5044 passed, 7 skipped, 1 failed. The failure is test/release-install-policy.test.ts (npm pack fails in my environment), and it fails the same way on 6c324d7.
  • tsc -p tsconfig.json --noEmit: the same 27 errors before and after; none in the changed files.
  • npm run docs:check: passes.
  • Not run against a live 0.26 engine.

@vercel

vercel Bot commented Nov 13, 2026

Copy link
Copy Markdown

@guibeira is attempting to deploy a commit to the rohitg00's projects Team on Vercel.

A member of the Team first needs to authorize it.

@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 70dadb18-b418-428a-a137-4d4834816431

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 3 potential issues.

Devin Review

Comment thread src/index.ts Resolved
Comment thread .env.example Resolved
Comment thread .env.example
# =============================================================================
# agentmemory configuration
# =============================================================================
#

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Commits retain prohibited attribution trailers

Both 0a056c7 and eb131ee contain Co-authored-by: iii harness (claude-opus-5-5). The repository prohibits AI attribution headers in commits.

Learn more

The repository's contribution rules prohibit attribution headers naming Claude in commit messages. Both new commit messages include a Co-authored-by trailer naming claude-opus-5-5; this is separate from their missing DCO sign-offs.

Example: The commit restoring this header ends with Co-authored-by: iii harness (claude-opus-5-5) <noreply@iii.dev>.

Recommended fix: Remove the prohibited trailers while amending the two commits.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

648883a and f608db8 no longer carry the attribution trailer. The follow-up fixup commit 0ec2a9b still does; the fixup commits will be squashed into f608db8 and the trailer removed before merge. Leaving this open until then.

devin-ai-integration[bot]

This comment was marked as resolved.

The merge in 678c67a joined the first header lines into
"# ====...# agentmemory configuration". Restore the four header lines
from main.

Signed-off-by: guibeira <contact@guibeira.dev>
…server and clear runtime timers

- An unreadable boot vector checkpoint was retried on every 30 s tick, and
  every retry rebuilt the whole BM25 index. The retry re-set its own
  "unavailable" flag, a paused recovery counted as a reason to rebuild, and
  the pause message in lastError read as a failed checkpoint save. Retries
  now back off from 30 s to 30 min, and an unusable checkpoint no longer
  triggers a keyword rebuild.
- A checkpoint that comes back with the wrong embedding dimension logged
  "Refusing to start" on every tick while the worker kept running. It now
  pauses recovery once (pauseForUnusableCheckpoint) and is not retried.
- The engine::on-workers-changed observer is created through
  Bindings.observeRuntime, so it gets the OBSERVER_ID_CONFLICT retry and
  appears in runtime.bindings, and it filters input.event to
  worker_connected and worker_disconnected in the engine.
- The duplicate-worker check runs at boot, when the state worker returns and
  on worker connect/disconnect events instead of polling
  engine::workers::list every 30 s. A disconnect now clears the warning.
- Shutdown clears the runtime tick and the compression sweep and stops the
  re-arm.
- Status and the 0.26 upgrade guide no longer say that delete cascades,
  relation reactions, failure audits and post-remember graph extraction
  run as direct calls; those observers have no direct-call path and are
  reported separately.

Tests boot the real src/index.ts main() against an in-memory engine
(test/helpers/fake-engine.ts) and drive the tick and SIGTERM handler it
installs.

Signed-off-by: guibeira <contact@guibeira.dev>
@guibeira
guibeira force-pushed the fix/iii026-observers-review branch from 2026280 to f608db8 Compare November 13, 2026 13:08
…nged observer and clear runtime timers

Run the duplicate-worker check as soon as the workers-changed observer
becomes active again, so a short outage between two ticks is covered, and
keep the tick check only while that observer is not active. A check now
publishes its result only when no check started after it has already
finished, so a late response cannot erase a newer warning.

Signed-off-by: guibeira <contact@guibeira.dev>
Co-authored-by: iii harness (claude-opus-5-5) <noreply@iii.dev>
devin-ai-integration[bot]

This comment was marked as resolved.

…nged observer and clear runtime timers

Treat a workers-changed conflict replacement that is already active as a
recovery, so the events missed during the conflict get one duplicate
check, and report no worker changes after stop(). Drop a comment that
only restated onRuntimeHandle.

Signed-off-by: guibeira <contact@guibeira.dev>
Co-authored-by: iii harness (claude-code/claude-opus-5-5) <noreply@iii.dev>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant