Repository navigation
v1.91.138.0 feat(eval-plan): cli-run stops a subscription eval at the first quota error - #3273
merit-blake wants to merge 8 commits into
Conversation
…uota stop gstack-eval-plan cli-run runs jobs as `claude -p --output-format json` children through a pull-based pool of --workers slots. The first quota or rate-limit result (a structured signal, a failed call's quota text, or exit 75 from the batch helper's open circuit) stops the run: no new job starts, unstarted jobs are recorded cancelled, and a quota is never retried. The child env drops the API billing routes. Records go to cli-run.jsonl, so a rerun resumes. A full run re-executes itself under the seat-pinning batch helper (GSTACK_CLAUDE_BATCH, default claude-batch) and runs its preflight inside; smoke scope runs at most 10 jobs on 2 workers without it. New result codes EVAL_QUOTA_STOP (exit 75), EVAL_BATCH_HELPER_MISSING and EVAL_BATCH_PREFLIGHT_FAILED live in lib/result-codes-eval-run.ts, spread into RESULT_CODES so lib/result-codes.ts stays under the module-size ratchet; the fixtures that copy that module's import closure list it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the run The /eval-plan template now says a subscription (CLI-metered) run goes through `gstack-eval-plan cli-run`, never a hand-written loop, and the first quota or rate-limit error ends it (EVAL_QUOTA_STOP): no retry, no seat switch, no API key, no fallback. Step 3 runs CLI-metered pilot attempts with `cli-run --scope smoke` and settles them `unknown`; Step 4 names `cli-run --scope full --class <standard|fable>` for the approved full run. The preregistration template's stop rules, docs/skills.md and the placeholder allowlist (<jobs.jsonl>, <standard|fable>) follow, plus a structure test for the template. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The helper and its preflight now see the same environment as the children, with the API billing routes removed, so nothing downstream of a quota stop can fall back to the API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…me binds prompt and args Outside review (Sol) findings: - A structured quota record on stderr, or a stream-json rate_limit_event with rate_limit_info.status=rejected, now stops the run at any exit code. Before, it classified as a retryable error. - Resume skips an ok record only when the job's prompt and args hash matches, so a rerun with a changed model arm is not silently skipped. - The output directory is 0700 and its files 0600, also on resume. - Tests seed every API billing variable and check the helper too, and pin that API-funded pilot attempts keep the project's own command. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… first quota error Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
|
Thanks @merit-blake. The incident write-up is excellent, and the core of this is right: a bounded pull pool, a stop on the first quota signal that never relies on
Everything else checked out on a merge with current main: typecheck, fresh generated docs, the size guard, parity, and 599 focused tests including your 41. The fake-claude quota run also behaved: 2 calls, 3 jobs cancelled, exit 75, the key unset in the child, 0700/0600 output, and a correct resume. Once these are in, we'll port it to a crew branch with your authorship so the paid gate can run with secrets. |
|
Thank you for this. A crew lane is picking it up to land on current main with Co-authored-by credit to you. |
|
Thank you @merit-blake. The idea at the heart of this PR shipped in v1.92.50.0, through #3751 and wave #3788: a seat-metered run should stop at its first quota or rate-limit error, with no retry. It's now a stop rule in the /eval-plan skill and its preregistration template, credited to you in the changelog. We wrote that rule as skill prose on current main and didn't merge this branch. The Closing this in favor of #3751. Thanks again for the idea. |
What
gstack-eval-plan cli-run: a gstack-owned driver for subscription (CLI-metered) evals that stops at the first quota error./eval-plannow routes subscription runs through it instead of telling agents to bring their own loop.Why
An agent-written eval driver ran 1,164 headless
claude -pcalls in about 4 hours on one Claude subscription seat. It queued every job in a thread pool up front, used--output-format text, ignored exit codes and stderr, and retried unparseable output twice. When the five-hour window ran out, 389 calls were rejected in 4.5 minutes and the loop kept firing. Each rejected call left a transcript record witherror: "rate_limit",apiErrorStatus: 429andquotaLimits.status: "rejected", whilerateLimitTypewas oftennull.How
--workers(1–8, default 2) pull-based slots. A job starts only when a slot is free.error: rate_limit, 429, a rejectedquotaLimitsstatus, a rejected streamrate_limit_event), the CLI's limit message on a failed call, or exit 75 all end the run. No new job starts, unstarted jobs are recordedcancelled, and the run exits 75 withEVAL_QUOTA_STOP. A quota is never retried;--retriescovers other errors only. Detection never relies onrateLimitType.okjobs whose prompt and args hash is unchanged.--scope fullre-runs itself underclaude-batch run --repo <repo> --class standard|fable -- …. That external helper pins one seat and trips a durable circuit on the first quota error (GSTACK_CLAUDE_BATCHoverrides the command). Inside the batch, the driver checks the marker env (MERIT_CLAUDE_BATCH_ID, overridable withGSTACK_CLAUDE_BATCH_MARKER), then runs<helper> preflightbefore the first call.EVAL_BATCH_HELPER_MISSING(exit 3), and nothing falls back to unwrapped calls.EVAL_BATCH_PREFLIGHT_FAILED.--scope smoke(at most 10 jobs, 2 workers) needs no helper.lib/result-codes-eval-run.ts, spread intoRESULT_CODESthe same way asresult-codes-release.ts. This keepslib/result-codes.tsunder the module-size limit. Each code has adocs/troubleshooting.mdanchor.Tests
test/eval-cli-run.test.ts(20 cases) drives the real bin against a fakeclaudeand a fakeclaude-batchon a minimal PATH. No real model call and no network. The fakeclaudekeeps its own in-flight counter under a lock, so concurrency is observed outside the runner. Coverage:rateLimitType, stderr-only records, streamrate_limit_event, and answers that merely mention "rate limit"Mutation checks:
Checks run
bun test test/eval-cli-run.test.ts: 20 pass, 0 fail.bun run typecheck: clean.bun run typecheck:test: no new diagnostics.bun run gen:skill-docsregeneratedeval-plan/SKILL.mdand the agents digest.bun run testdid not pass on this machine: 434 failures on this branch, and 634 on base 952f95a. The failures are in PTY, browser, Docker, pregate and install tests. The files this change plausibly touches fail identically on HEAD and base when run alone (289 pass / 24 fail both). CI is the acceptance run.Follow-ups
ask: Once the externalclaude-batchhelper is installed, run one--scope smokeand one tiny--scope fullagainst it to confirm the live contract (marker, preflight, exit 75 propagation).session: gstack-contrib-eval-quota-stop