Concurrent sessions retry a failing provider independently — no shared backoff or circuit breaker (2.0) #55075

Description

Follow-up to #52513 (closed during the 1.x → 2.0 transition, with an invitation to re-open if this still applies to 2.0 — it does). Originally reported against 1.18.x; I've now verified at the code level that 2.0.28 still has no shared per-provider retry/health state: each session runs its own retry policy against a failing provider, so N concurrent sessions keep hitting the same failing endpoint at overlapping times, and every retry publishes session-scoped events to all connected clients.

What changed in 2.0 (all refs at tag v2.0.28, packages/core/src/session/runner/):

• the per-session policy grew: exponential 2s capped at 10s per gap, up to 10 retries per cycle — roughly 84s of waiting per exhausted cycle (67–101s with jitter): retry.ts#L45
• it is still scoped to a single session: policy(sessionID) closes over its own attempt counters (retry.ts#L64), and the runner instantiates it per step run with SessionRunnerRetry.make(bus, sessionID) (llm.ts#L209)
• Retry-After is honored per session too, capped at 15 min (retry.ts#L32)
• there is no provider-level shared state anywhere (no health tracking, no breaker) — so nothing coordinates concurrent sessions against the same failing provider

To be fair, the per-session side improved nicely in 2.0: bounded retries, Retry-After honored with a sane cap. The missing piece is still coordination between sessions.

Net effect on 2.0: during a provider outage/limit window, 5 concurrent sessions produce ~50 futile attempts spread over ~1.5 minutes per exhausted cycle, repeating as long as the provider is down and parents re-dispatch — the same storm as 1.x, just longer.

Behavioral evidence from 1.18.x (mechanism unchanged in 2.0): 2,610 stream error entries over ~7 weeks of heavy parallel use, worst single day 864, one observed minute with 5+ concurrent sessions failing every 2–5 s against the same provider.

Suggested direction — unchanged from #52513: shared state keyed by provider (+ endpoint/credential) in front of the per-session policy:

• 429: keep honoring Retry-After (2.0 already caps it) but share the resulting next-allowed time across sessions
• 5xx / connection errors: circuit breaker (consecutiveFailures, closed/open/half-open, one half-open probe), opening after ~one exhausted policy (~84s on 2.0)
• while open: one deduplicated provider.status event instead of per-session retry events; in-flight streams finish naturally, only new attempts are gated
• first pass: share state within one server process

Happy to attempt a PR — a small provider-health module next to packages/core/src/session/runner/ looks like a natural home, but I'd love a pointer to where you'd want it.

Plugins

None

OpenCode version

2.0.28

Steps to reproduce

1. Configure a provider that returns 429/503 (or point it at a mock server that does).
2. Start one parent session and 3–5 parallel subagents using that provider.
3. Watch provider traffic and client events while the provider is failing.

Expected: a shared backoff/circuit opens; only one probe request in flight; one throttled provider status event.
Actual: each session retries on its own schedule (up to 10 attempts, ~84s per cycle on 2.0.28); attempts overlap; clients receive per-session retry events from every session.

Operating System

Windows 11 (build 26200)