Skip to content

fix(server): Claude pool limits keep the earliest gateway reset - #3

Merged
Ali-Al1 merged 4 commits into
base/upstream-main-920ffcdd04-hosted-cifrom
fix/cliproxy-pool-limited-reset
Nov 9, 2026
Merged

Ali-Al1 merged 4 commits into
base/upstream-main-920ffcdd04-hosted-cifrom
fix/cliproxy-pool-limited-reset

Conversation

@Ali-Al1

@Ali-Al1 Ali-Al1 commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

Problem

When every account in a CLIProxyAPI multi-provider pool (mimo-v2.6-flash-all-pool, GLM/DeepSeek pools) is exhausted, T3 should enter limited with the real reset of whichever account resets first. The Claude adapter only read a reset from (reset after …) text, and only in two exact places: directly after a 429's API Error: prefix, or in 9router's wrapped 503 shape with the text right after the JSON. CLIProxyAPI's model_cooldown body carries the earliest reset as reset_seconds, which T3 ignored, so pool limits failed as provider errors or as usage limits with no reset.

Change

providerFailureFromResult in ClaudeAdapterV2.ts:

  • One shared reset parser: the first (reset after 1h2m3s) in the result, otherwise "reset_seconds":N (raw or JSON-escaped). Reset = result arrival + seconds. The same bounds as before apply (more than 0 and representable).
  • The pool's own reset is the first (reset after …) match, so in … cooling down (reset after 2h54m21s) (last error: GoUsageLimitError (reset after 9h0m0s)) T3 uses 2h54m21s, not 9h. If the first match is (reset after 0s), T3 no longer falls back to reset_seconds. A reset_seconds inside (last error: …) is ignored, because it belongs to one credential, not the pool.
  • 429: reads the reset from either form when the result is an error (is_error or an API Error: prefix). The class is still usage_limit, as before.
  • Wrapped gateway 503 (API Error: 503 [x] [429]: {"type":"error","error":{"type":"rate_limit_error",…): the reset can now be inside the JSON message (where the CLIProxyAPI patch puts it) or come from reset_seconds.
  • "code":"model_cooldown" on any status, with a reset: usage_limit with resetAt, retryable.
  • auth_not_found / no auth candidates, and anything else with no reset signal, stays provider_error with no reset. A 401 or overloaded_error wrap still stays provider_error.

Tests

ClaudeAdapterV2.test.ts adds 16 cases ("reads a credential pool's earliest reset from …"):

  • 429s: (reset after 2h54m21s), a reset outside the API Error: prefix, model_cooldown with reset_seconds (raw and JSON-escaped), and a Claude rate_limit_error with reset_seconds.
  • A 500 model_cooldown, and wrapped 503s with the reset in the message or in reset_seconds.
  • A zero reset_seconds, and auth_not_found (plain text and JSON), which stay provider_error.
  • The proxy's real 429 shape, with both the pool's reset and a later reset in its last error, in the raw API Error: 429 {…rate_limit_error…} form and the rendered API Error: Request rejected (429) · … form. Both give 2h54m21s.
  • (reset after 0s) with a reset_seconds in the last error gives no reset. A model_cooldown whose only reset_seconds is in its last error stays provider_error. Both fail without the r1 parser change.

The existing gateway, direct-429 and buffered-wake cases are unchanged and still pass.

  • vp test run src/orchestration-v2/Adapters/ClaudeAdapterV2.test.ts: 361 passed, 1 skipped (with AgentContextBudget.test.ts: 363 passed)
  • vp run check:changed: format, lint, knip, graphql and typecheck pass. Its root test step fails on Windows before running any test: the web dependency optimizer cannot load a path containing Asus%20X13. This is a local path problem, unrelated to this change.
  • Related server integration tests: ClaudeReplayFixtures.integration.test.ts has 2 Bash-streaming failures that also happen on unmodified upstream/main on this machine. Everything else passes.

Base

This targets base/upstream-main-920ffcdd04-hosted-ci, which is pingdotgg/t3code main at 920ffcd plus one CI-only commit (b5d29a8). That commit ports the fork's hosted-runner fallback from #2 to ci.yml and mobile-fingerprint-check.yml: github.repository_owner == 'pingdotgg' && 'blacksmith-…' || 'ubuntu-24.04' (or macos-26). The earlier base, base/upstream-main-920ffcdd04, requested only Blacksmith runners, which the fork doesn't have, so CI stayed queued forever. The fork's main predates this adapter code. The PR diff is still only the two adapter files.

CI on hosted runners

Run 42550208059 ran on GitHub-hosted ubuntu-24.04 / macos-26 runners. ClaudeAdapterV2.test.ts passed on Test Server 5 (362 tests). Lint, Build and Typecheck, Test, Rust, Release Smoke, Mobile Native Static Analysis, Test Server 2–8 and Native fingerprint diff also passed. Check failed because of two problems this PR doesn't touch:

  • Test Server 1: storageCleanup.test.ts > compares the checked-out branch without a full status gets outcome: "failed" ("gitfile does not point to a valid repository"), not the "kept" it expects. The runner has git 2.56.0. This is upstream snapshot code.
  • Test Web: hit the job's 10-minute timeout with no failing test. The GitHub-hosted runner is slower than the 8-vCPU Blacksmith runner the timeout was set for.

Review log

  • Opus r1: code approved. Actions taken:
    • A (high): CI could never finish on the snapshot base. Moved the PR to the hosted-CI base above.
    • B (low): added the proxy-shaped 429 rows with a later last-error reset (raw body and rendered forms).
    • C (info): done. A first match of (reset after 0s) no longer falls back to reset_seconds, and a reset_seconds inside (last error: …) is ignored. Two rows cover this.

🤖 Generated with Claude Code

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:M labels Nov 9, 2026
Ali-Al1 and others added 2 commits November 9, 2026 02:06
When every account in a CLIProxyAPI pool is cooling down, the proxy reports
the earliest account reset either as "(reset after 1h2m3s)" text or as a
model_cooldown body with reset_seconds. The Claude adapter only read the
text form, and only directly after a 429's "API Error:" prefix or in 9router's
exact wrapped 503 shape, so pool limits failed as provider errors or as
usage limits with no reset.

Read the reset from either form on a 429, a wrapped gateway 503, or any
model_cooldown body, and class a model_cooldown with a reset as a usage
limit whatever status carries it. auth_not_found and other errors with no
reset stay provider errors.

Co-Authored-By: Claude Code <noreply@anthropic.com>
CLIProxyAPI appends the last upstream error to an exhausted pool's 429,
e.g. "(reset after 2h54m21s) (last error: GoUsageLimitError (reset after
9h0m0s))". The pool's own reset is the first match. When that first match
is "(reset after 0s)", or the pool names no reset, the parser no longer
falls back to a reset_seconds quoted inside "(last error: ...)": that is
one credential's reset, not the pool's.

Adds the proxy-shaped 429 in its raw Claude body and rendered forms, and
both last-error fallback cases, to the pool reset table.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@Ali-Al1
Ali-Al1 force-pushed the fix/cliproxy-pool-limited-reset branch from 15fdada to dc30d01 Compare November 9, 2026 02:18
@Ali-Al1
Ali-Al1 changed the base branch from base/upstream-main-920ffcdd04 to base/upstream-main-920ffcdd04-hosted-ci November 9, 2026 02:18
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire — 5.0 KiB — 6.8 KiB ✅
Codex Thread snapshot wire — 3.8 KiB — 4.9 KiB ✅
Codex Live turn WebSocket wire — 1.2 KiB — 2.0 KiB ✅
Codex Live turn WebSocket decoded — 21.5 KiB — 24.4 KiB ✅
Codex Live turn messages — 1 — 8 ✅
Claude Total thread wire — 5.0 KiB — 6.8 KiB ✅
Claude Thread snapshot wire — 3.8 KiB — 4.9 KiB ✅
Claude Live turn WebSocket wire — 1.2 KiB — 2.0 KiB ✅
Claude Live turn WebSocket decoded — 21.9 KiB — 24.4 KiB ✅
Claude Live turn messages — 1 — 8 ✅

Baseline: unavailable · PR result: e956f66 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@Ali-Al1

Ali-Al1 commented Nov 9, 2026

Copy link
Copy Markdown
Owner Author

Sol review (9 Nov 2026) — HAPPY on code

Reviewed production delta at head dc30d01a55 vs origin/base/upstream-main-920ffcdd04-hosted-ci (ClaudeAdapterV2.ts only; tests are the second file).

  • First (reset after …) wins; pool suffix is ordered before (last error: …) by the CPA half, so the earliest positive-weight reset is the one T3 keeps.
  • reset_seconds is ignored inside (last error: …) and is not used when the first (reset after …) is 0s (finding C behaviour).
  • 503 gateway path still requires the wrapped rate_limit_error shape before treating the body as a pool/gateway cooldown; plain auth_not_found stays provider_error.
  • 429 (or is_error / API Error:) still yields usage_limit with resetAt when parseable; non-429 model_cooldown with a real reset_seconds also becomes usage_limit + resetAt + retryable.
  • Tests cover the production shapes (raw 429 body, rendered Request rejected (429), first-match vs later 9h last error, 0s, non-429 cooldown, auth_not_found).

No Sol blocking findings. Delivery note (Opus r2, accepted): production path is the CLIProxyAPI carry-forward half; this PR is hardening for an upstream PR to pingdotgg/t3code (needs Ali — public) or a local build. Fork merge alone does not deploy the nightly.

Review log: Opus r1→author→r2 happy; Sol happy; DeepSeek trial pending on the CLIProxyAPI half (separate artifact).

🤖 Generated with Claude Code

@Ali-Al1

Ali-Al1 commented Nov 9, 2026

Copy link
Copy Markdown
Owner Author

Re-opening to re-run CI against base base/upstream-main-920ffcdd04-hosted-ci now that the git 2.56 gitfile classifier fix (#4) has merged.

@Ali-Al1 Ali-Al1 closed this Nov 9, 2026
@Ali-Al1 Ali-Al1 reopened this Nov 9, 2026
Empty commit: same tree as dc30d01 so GitHub rebuilds pull/3/merge
against base/upstream-main-920ffcdd04-hosted-ci @ c45fdeb (gitfile
classifier fix from #4).

Co-Authored-By: Claude Code <noreply@anthropic.com>
@Ali-Al1

Ali-Al1 commented Nov 9, 2026

Copy link
Copy Markdown
Owner Author

Opus review round 2 (9 Nov 2026, head a2ed31ba) — HAPPY

Confirmed all r1 actions in the diff: A base is the hosted-CI base; B proxy-shaped 429 rows (raw rate_limit_error and rendered Request rejected (429) ·) with a later 9h last-error reset both expecting 2h54m21s; C (reset after 0s) gives no fallback to reset_seconds, and a reset_seconds inside (last error: …) is cut off by the text.slice(0, lastError) step with a covering row. Parser order, bounds, model_cooldown → usage_limit+resetAt (any status; provider_error when no reset), auth_not_found → provider_error, wrapped-503 prefix gating, 401/overloaded paths all verified. Diff stays exactly 2 files.

Findings:

  • Low 1: a real model_cooldown raw body with a cause loses its top-level reset_seconds because Go map key order puts message (ending (last error: …)) before reset_seconds, and the text is cut at (last error:. Not reachable in production today — CPA's Claude handler sends only error.message, and the live 429 path works. Hardening for later, not a blocker.
  • Low 2: unpatched upstream CPA appends (reset after X) after (last error: …), so a cause's later reset could win by first-match. Ali's patched CPA 8.0.60 puts the pool reset first (verified live 9 Nov), so no effect on this setup. Relevant only if/when an upstream PR is opened — notes for that PR.
  • Info: some new rows also pass on base as regression checks (the "both fail without r1" claim is vs r0); merging into the fork deploys nothing to the nightly — live use needs the upstream PR (needs Ali's approval) or a local build.

Verdict: happy — findings 1–2 are post-merge hardening, not blockers.

@Ali-Al1

Ali-Al1 commented Nov 9, 2026

Copy link
Copy Markdown
Owner Author

DeepSeek trial review r1 (9 Nov 2026) — HAPPY

Read-only review at head a2ed31ba: scope verified (exactly 2 files, +218/−23; retrigger commit empty). Parser regexes copied verbatim into a standalone script and run against ~25 shapes; cross-checked against the deployed CPA patch (modelCooldownError appends ResetAfterHint() before (last error: …), empty when ceil(resetIn)≤0; JSON order code, message, model, reset_seconds). Did not run vitest locally; CI green taken as given.

  • High: none.
  • Medium M1: the (last error: …) truncation applies only to the reset_seconds fallback, not the (reset after …) text scan (ClaudeAdapterV2.ts:4111-4120). (1) Pool with empty hint but last error carrying (reset after 9h0m0s) → T3 takes 9h as the pool reset (reproduced: 32400). (2) (reset after 0s) inside last error short-circuits before the reset_seconds fallback → valid pool reset_seconds:600 discarded (reproduced). Production CPA 8.0.60 skirts this only while the hint is present and placed first; hint-stripped or zero-delay cooldowns expose it. Suggested: apply the last-error cut to the text scan too so both branches obey one rule.
  • Low L1: top-level reset_seconds unreachable whenever the message contains (last error: (real CPA JSON orders message before reset_seconds) — same root as Opus r2 Low 1; same fix should resolve.
  • Low L2: \?" in GATEWAY_RESET_SECONDS can't distinguish nesting level; load-bearing only via the truncation heuristic. Not reachable from CPA today.
  • Low L3: test table has no row where (reset after …) inside (last error: …) is the first/only text match (pool-no-hint and 0s variants); rows don't assert failure.code.
  • Info: bounds correct; first-match correct for the documented CPA shape; auth_not_found/provider_error paths correct; model_cooldown marker risk negligible; wrapped-503 failureHint === undefined asymmetry pre-existing.

Verdict: happy — M1 is a real gap in the PR's own rule, worth a follow-up or an in-round fix at Opus's discretion; does not block the documented production path.

Opus judging M1–L3 next; judgment to be recorded here before merge.

@Ali-Al1

Ali-Al1 commented Nov 9, 2026

Copy link
Copy Markdown
Owner Author

Opus judging DeepSeek r1 (9 Nov 2026) — one fix required, then merge

Finding Ruling Reason
M1 Fix in this PR Reachable in production: CPA strips the pool hint when status≠429 or Retry-After unparseable, so the last error's (reset after 9h0m0s) becomes T3's reset (9h-late resume), and (reset after 0s) in the last error discards a valid top-level reset_seconds. Small local fix.
L1 (with r2 Low 1) Fix in this PR (same change) Scrubbing only the last-error segment instead of cutting everything after it fixes top-level reset_seconds reachability.
L2 Decline No CPA output today nests reset_seconds where \?" misreads; the segment rule removes the realistic case.
L3 Fix in this PR (tests for M1) Required rows come with M1 anyway; asserting failure.code is cheap.
Info No action Wrapped-503 failureHint asymmetry pre-existing.
r2 Low 2 (upstream ordering) Decline as follow-up Only matters for a future upstream CPA PR; Ali's 8.0.60 puts the hint first (verified live).

Spec for the fix (gatewayCooldownResetAt): scrub the text once via withoutLastError(text) — from each (last error: remove up to (not including) the first unescaped " (end of the enclosing JSON message string, so top-level ,"reset_seconds":… survives); no quote → remove to end; repeat. Run GATEWAY_RESET_AFTER first, then GATEWAY_RESET_SECONDS, both on the scrubbed text; delete the old slice(0, lastError). Keep first-match order and bounds. Test rows: no-hint + 9h-in-last-error (raw + rendered), no-hint + 0s-in-last-error + top-level reset_seconds:600, real Go field order code, message, model, reset_seconds; all table rows assert failure.code; existing rows must pass unchanged.

Implementation delegated to Opus now; re-check of its diff, then merge — no new full Opus round needed.

A gateway's "(last error: …)" quotes one credential's error, not the pool's
earliest reset. The "(reset after …)" scan read the whole result while the
reset_seconds fallback was cut at the marker, so a hint inside the last error
was taken as the pool's reset and a top-level reset_seconds after the message
was dropped. Remove each last error up to the end of the JSON string holding
it (or the end of rendered text) and run both scans on what remains.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@Ali-Al1

Ali-Al1 commented Nov 9, 2026

Copy link
Copy Markdown
Owner Author

Fix re-check (Sol, 9 Nov 2026) — matches Opus's spec

Commit e956f66d (+70/−4 on top of a2ed31ba; PR still exactly 2 files).

  • withoutLastError: removes each (last error: segment up to (keeping) the first unescaped " that closes the enclosing JSON message string — so top-level ,"reset_seconds":… survives; unescaped quote search skips \"; no quote → remove to end (rendered text); repeats until no marker remains. Exactly as specified.
  • gatewayCooldownResetAt: one scrub, GATEWAY_RESET_AFTER first, GATEWAY_RESET_SECONDS fallback, both on the scrubbed text; old slice(0, lastError) gone; regexes, first-match order, resetAtAfterSeconds bounds and zero handling unchanged; L2 nesting limitation left as declined.
  • Tests: every table row now asserts failure.code (real api_error_<status> codes, with the existing class assertion keeping the usage_limit/provider_error split — correct adaptation of the brief); 4 new rows (9h-in-last-error raw + rendered, 0s-in-last-error + top-level reset_seconds:600, real Go field order …, "reset_seconds":3723); 365 passed / 1 skipped with the fix; exactly those 4 rows fail with production reverted; all pre-existing rows pass both ways.

No Sol findings. CI run 42591338698 in progress — merge on green (per Opus: no new full round needed after this re-check).

@Ali-Al1
Ali-Al1 merged commit 4614f68 into base/upstream-main-920ffcdd04-hosted-ci Nov 9, 2026
27 checks passed
@Ali-Al1
Ali-Al1 deleted the fix/cliproxy-pool-limited-reset branch November 9, 2026 09:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant