Observer wedges permanently on a stale seven_day quota entry that no incoming event can refresh #5012

Summary

The observer's rate-limit cache is keyed by the event's top-level rateLimitType and has no TTL. The Claude Code CLI now reports per-window utilisation nested under rate_limit_info.unifiedWindows, and only names one window at the top level — the binding one. unifiedWindows is never read by claude-mem (the string does not appear in worker-service.cjs at all).

Consequence: once a seven_day entry is cached at a high utilisation, nothing can ever overwrite it, because subsequent events are keyed five_hour. The quota guard iterates all windows on every event, so each fresh five_hour event re-reads the stale seven_day value and aborts the session. That arms a 30-minute cooldown; the probe re-aborts on the same entry; repeat forever.

In my case the observer was wedged for ~41 hours (2026-10-21T21:17:46Z → manual intervention 2026-10-23T14:57Z) while actual seven-day utilisation was 16%. It would not have self-cleared until resetsAt (2026-10-26) or a worker restart.

Impact

• Memory capture stops completely and does not recover on its own.
• The failure is more likely the longer the worker runs — mine had 167h uptime, which I don't think is unusual.
• The user-facing message is actively misleading (see below), so the natural response is to go buy capacity or switch providers, neither of which helps.

The message is wrong, which is most of the damage

The outage banner and observer-health.json both say:

Provider reported the inference allowance exhausted

The provider reported no such thing. The cached entry that triggers the abort has:

"status": "allowed_warning"

This is a locally inferred threshold abort (utilization >= 0.93), not a provider rejection — the code path for a real rejection is a separate branch (status === "rejected"). The banner also states:

Restarting will NOT help here, and it clears the backoff that is currently keeping claude-mem from hammering the provider

For this failure mode that is inverted: restarting is the only fix, and the backoff is throttling traffic that is entirely within quota. I'd suggest the wording distinguish a provider rejection from a local threshold abort, and only claim "allowance exhausted" for the former.

Evidence

GET http://127.0.0.1:37777/api/health → rateLimits, taken while wedged:

{
  "five_hour": {
    "status": "allowed",
    "rateLimitType": "five_hour",
    "resetsAt": 1792731000,
    "unifiedWindows": {
      "five_hour": { "utilization": 0.26, "resetsAt": 1792731000 },
      "seven_day": { "utilization": 0.16, "resetsAt": 1793034000 }
    },
    "observedAt": 1792727400073
  },
  "seven_day": {
    "status": "allowed_warning",
    "rateLimitType": "seven_day",
    "utilization": 0.97,
    "resetsAt": 1793034000,
    "surpassedThreshold": 0.75,
    "unifiedWindows": {
      "five_hour": { "utilization": 0,    "resetsAt": 1792635000 },
      "seven_day": { "utilization": 0.97, "resetsAt": 1793034000 }
    },
    "observedAt": 1792617466708
  }
}

Two things stand out:

1. The seven_day entry's observedAt is 2026-10-21T21:17:46.708Z — frozen at the exact second capture stopped, two days earlier. Its resetsAt is still in the future, so the staleness check never drops it.
2. The fresh five_hour entry carries the correct current value — unifiedWindows.seven_day.utilization: 0.16 — in the very same object. The right answer is already in hand and simply isn't read.

Resulting log, repeating every 30 minutes across days:

[2026-10-23 14:19:32.982] [WARN] [SDK] Aborting session for quota guard:
  quota:seven_day utilization 97.0% >= 93% {window=seven_day,
  authMethod=Claude Code OAuth token (read from system keychain at spawn)}
[2026-10-23 14:19:41.275] [WARN] [SESSION] Generator paused for quota;
  preserving buffered work {pendingCount=318}
[2026-10-23 14:32:00.675] [WARN] [SESSION] Skipping generator start while the
  provider quota cooldown is active {window=seven_day, retryInMs=1060585}

Independent ground truth — spawning the same CLI by hand and reading the raw event:

{"type":"rate_limit_event","rate_limit_info":{
  "status":"allowed","rateLimitType":"five_hour","resetsAt":1792731000,
  "unifiedWindows":{
    "five_hour":{"utilization":0.22,"resetsAt":1792731000},
    "seven_day":{"utilization":0.16,"resetsAt":1793034000}}}}

No top-level utilization field at all, and seven_day appears only inside unifiedWindows.

Root cause

Three behaviours combine:

1. Cache keyed by top-level rateLimitType, no TTL. The store writes entries.set(info.rateLimitType ?? "default", {...info, observedAt: Date.now()}). observedAt is recorded but never consulted.
2. The guard iterates every window on every event. It walks ["five_hour","seven_day_opus","seven_day_sonnet","seven_day","overage"] and returns on the first breach. Its only staleness defence is if (resetsAt !== undefined && resetsAt <= now) continue — useless for a long window whose reset is days away. (If resetsAt were ever absent, the entry would be immortal.)
3. unifiedWindows is never parsed. Grepping the shipped bundle for unifiedWindows returns no match, so non-binding windows can only be written when the CLI happens to make them binding.

So seven_day is written exactly once — during a genuine spike — and is then pinned at that value for the life of the process.

Reproduction

1. Drive seven-day utilisation above 93% so the CLI emits a top-level rateLimitType: "seven_day" event. The observer correctly aborts here.
2. Let utilisation fall well below the threshold (new 5-hour windows, lighter use). The CLI now emits rateLimitType: "five_hour" with the true seven_day figure nested in unifiedWindows.
3. Observe the observer stay wedged indefinitely: every event re-trips the guard on the stale entry. It recovers only on worker restart or when seven_day.resetsAt passes.

Steps 2–3 are what I observed directly over ~41h. Step 1 is reconstructed from the logs and the cached entry's observedAt rather than deliberately re-driven, so treat the trigger as well-evidenced but not re-created from scratch.

Suggested fixes

1. Parse unifiedWindows and upsert every window it contains, not just rateLimitType. This alone resolves it — each five_hour event carries a current seven_day figure, so the stale entry is overwritten within seconds. Highest value, smallest change.
2. Use the observedAt you already record. Treat an entry older than some TTL (a few minutes) as unknown rather than authoritative. Good defence in depth if the payload shape shifts again.
3. Clear cached state on a successful probe. Right now a probe that succeeds doesn't invalidate the entry that caused the cooldown, so recovery depends entirely on the stale value disappearing by itself.
4. Fix the messaging (see above): don't report "Provider reported the inference allowance exhausted" for a locally inferred threshold abort, and don't tell users a restart won't help when it's the only remedy.

A smaller related observation: overageStatus: "rejected" with overageDisabledReason: "org_level_disabled" is permanent for accounts with extra usage switched off. The guard aborts on overageStatus === "rejected" when the window key is overage; if the CLI ever emits a top-level rateLimitType: "overage" for such an account, that would wedge the same way and never clear. I did not reproduce this one.

Workaround

Order matters. The worker loads quota-cooldown.json into memory once at boot, so deleting it afterwards has no effect — my first restart inherited the cooldown and stayed stuck:

rm -f ~/.claude-mem/quota-cooldown.json
curl -X POST http://127.0.0.1:37777/api/admin/restart

Verify with STORED | obsIds=[...] in ~/.claude-mem/logs/, not with the banner. Note observer-health.json keeps showing cooldownActive: true until the cooldown map next changes, so it reads stale even after a successful recovery. The SQLite-backed queue survives the restart.

Environment