Skip to content

fix: skip summarize for noop provider behind resilient wrapper (#996) - #1001

Closed
costajohnt wants to merge 1 commit into
rohitg00:mainfrom
costajohnt:fix/996-zero-llm-summarize-metrics
Closed

costajohnt wants to merge 1 commit into
rohitg00:mainfrom
costajohnt:fix/996-zero-llm-summarize-metrics

Conversation

@costajohnt

@costajohnt costajohnt commented Jul 2, 2026 •

Copy link
Copy Markdown

Summary

In zero-LLM mode mem::summarize records a failure in the health metrics on every invocation, so functionMetrics for mem::summarize shows an ever-growing failureCount (observed 1419 -> 1429 within minutes) even though skipping is the correct behavior without an LLM.

One correction to the diagnosis in the issue. The no_provider early return itself records nothing, so it is not the skip being tallied. The real problem is that the skip guard never fires at all. createProvider always wraps the base provider in ResilientProvider, whose name is resilient(noop), so the guard if (provider.name === "noop") in registerSummarizeFunction is dead code. Execution falls through to NoopProvider.summarize(), which returns an empty string, and the empty_provider_response branch then calls metricsStore.record("mem::summarize", latencyMs, false). That happens twice per Claude Code turn (the Stop hook posts both /agentmemory/summarize and /agentmemory/session/end), which matches the observed growth rate.

Fix

  • Add an optional isNoop flag to MemoryProvider and set it on NoopProvider.
  • Propagate the flag through the wrappers. ResilientProvider mirrors its inner provider, and FallbackChainProvider is noop only when every chain member is.
  • Guard the summarize skip on provider.isNoop instead of the name. The skip returns no_provider before any metrics call, so zero-LLM invocations stay metrics-silent.

No new "skipped" metrics outcome is needed. Once the guard fires, the skip path records nothing, which keeps failureCount reserved for genuine failures.

Testing

  • New test/zero-llm-summarize-metrics.test.ts covers four cases: resilient(noop) skips with no_provider and never touches the metrics store, resilient(fallback(noop)) does the same, and a real provider (bare or in a chain) is not skipped and still records success metrics. The first two fail on main (they get empty_provider_response and a recorded failure) and pass with the fix.
  • Full npm test passes (1414 tests; the only local exceptions were 5 hook-project tests that assert on the checkout directory name and fail in any git worktree, unrelated to this change). npm run build and npm run skills:check also pass.

Fixes #996

Notes for reviewers

  • isNoop is optional with strict === true checks in the wrappers, so a provider that never sets it can only fail toward "not noop" (real work still happens). Making it a required member was considered and skipped to keep the diff minimal; happy to change if you prefer the compiler-enforced version.
  • Adjacent gap, not touched here: the viewer's Summarize button shows "Done" without inspecting the result, so in zero-LLM mode a user gets "Done" with no summary. Pre-existing behavior; worth a small follow-up if you want the button to surface the no_provider reason.

Summary by CodeRabbit

  • New Features

    • Added support for detecting “no LLM” provider chains more reliably, including wrapped and fallback provider setups.
    • Summarize requests now correctly skip LLM-dependent work when the active provider is effectively noop.
  • Bug Fixes

    • Fixed cases where provider detection could incorrectly treat wrapped noop providers as real providers.
    • Prevented metric recording when summarize cannot run due to a noop-only provider chain.
  • Tests

    • Added coverage for zero-LLM behavior across direct, wrapped, and fallback provider combinations.

The zero-LLM guard in mem::summarize checked provider.name === "noop",
but the provider is always wrapped in ResilientProvider whose name is
resilient(noop), so the guard never fired. Execution fell through to the
noop provider's empty response and recorded a metrics failure on every
invocation, inflating failureCount indefinitely in zero-LLM mode.

Add an isNoop flag to MemoryProvider, set it on NoopProvider, propagate
it through ResilientProvider and FallbackChainProvider, and guard
summarize on it. The no_provider skip stays metrics-silent.

Signed-off-by: John Costa <costajohnt@gmail.com>
@vercel

vercel Bot commented Jul 2, 2026

Copy link
Copy Markdown

@costajohnt is attempting to deploy a commit to the rohitg00's projects Team on Vercel.

A member of the Team first needs to authorize it.

@coderabbitai

coderabbitai Bot commented Jul 2, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b40e66b1-5bdf-41b8-8eab-42787781ffe2

📥 Commits

Reviewing files that changed from the base of the PR and between 93ae9bc and b8cf311.

📒 Files selected for processing (6)
  • src/functions/summarize.ts
  • src/providers/fallback-chain.ts
  • src/providers/noop.ts
  • src/providers/resilient.ts
  • src/types.ts
  • test/zero-llm-summarize-metrics.test.ts

📝 Walkthrough

Walkthrough

Adds an optional isNoop flag to the MemoryProvider interface, implemented by NoopProvider (always true), ResilientProvider (propagated from inner provider), and FallbackChainProvider (true only if all wrapped providers are noop). The mem::summarize handler's bailout check switches from provider.name === "noop" to provider.isNoop. A new test suite validates metrics recording behavior across these provider compositions.

Changes

isNoop propagation and summarize guard

Layer / File(s) Summary
MemoryProvider isNoop contract and implementations
src/types.ts, src/providers/noop.ts, src/providers/resilient.ts, src/providers/fallback-chain.ts
Adds optional isNoop?: boolean to MemoryProvider; NoopProvider sets it to true; ResilientProvider propagates it from its inner provider; FallbackChainProvider sets it to true only when every wrapped provider is noop.
Summarize handler noop guard update
src/functions/summarize.ts
Replaces the provider.name === "noop" check with provider.isNoop in the mem::summarize bailout, updating comments to explain wrapped noop scenarios.
Zero-LLM summarize metrics tests
test/zero-llm-summarize-metrics.test.ts
New test suite with mocked SDK, KV, and metrics store validating that summarize skips and avoids recording metrics when the effective provider chain is entirely noop (directly or via ResilientProvider/FallbackChainProvider), and succeeds with metrics recorded otherwise.

Estimated code review effort: 2 (Simple) | ~15 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant SummarizeHandler as mem::summarize handler
  participant Provider as MemoryProvider (wrapped)
  participant MetricsStore

  Caller->>SummarizeHandler: invoke mem::summarize
  SummarizeHandler->>Provider: check provider.isNoop
  alt provider.isNoop is true
    Provider-->>SummarizeHandler: isNoop = true
    SummarizeHandler-->>Caller: success:false, error:"no_provider"
  else provider.isNoop is false
    Provider-->>SummarizeHandler: isNoop = false
    SummarizeHandler->>Provider: run summarize
    Provider-->>SummarizeHandler: summary result
    SummarizeHandler->>MetricsStore: record(mem::summarize metric)
    SummarizeHandler-->>Caller: success:true
  end
Loading

Possibly related PRs

  • rohitg00/agentmemory#791: Both PRs modify the mem::summarize handler in src/functions/summarize.ts, touching overlapping logic in the summarize flow.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly states the core change: skipping summarize for a noop provider behind a resilient wrapper.
Linked Issues check ✅ Passed The changes implement #996 by propagating noop detection and skipping mem::summarize without recording false failures.
Out of Scope Changes check ✅ Passed All code changes support the zero-LLM summarize fix and its tests; no unrelated scope is evident.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@costajohnt costajohnt closed this Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Zero-LLM mode: mem::summarize no-op skip (no_provider) inflates failureCount with false failures

1 participant