Skip to content

v1.91.138.0 feat(eval-plan): cli-run stops a subscription eval at the first quota error - #3273

Closed
merit-blake wants to merge 8 commits into
garrytan:mainfrom
merit-blake:eval-quota-stop
Closed

merit-blake wants to merge 8 commits into
garrytan:mainfrom
merit-blake:eval-quota-stop

Conversation

@merit-blake

Copy link
Copy Markdown
Contributor

What

gstack-eval-plan cli-run: a gstack-owned driver for subscription (CLI-metered) evals that stops at the first quota error. /eval-plan now routes subscription runs through it instead of telling agents to bring their own loop.

Why

An agent-written eval driver ran 1,164 headless claude -p calls in about 4 hours on one Claude subscription seat. It queued every job in a thread pool up front, used --output-format text, ignored exit codes and stderr, and retried unparseable output twice. When the five-hour window ran out, 389 calls were rejected in 4.5 minutes and the loop kept firing. Each rejected call left a transcript record with error: "rate_limit", apiErrorStatus: 429 and quotaLimits.status: "rejected", while rateLimitType was often null.

How

  • Bounded pool. Exactly --workers (1–8, default 2) pull-based slots. A job starts only when a slot is free.
  • Quota stop. Structured signals on stdout or stderr (error: rate_limit, 429, a rejected quotaLimits status, a rejected stream rate_limit_event), the CLI's limit message on a failed call, or exit 75 all end the run. No new job starts, unstarted jobs are recorded cancelled, and the run exits 75 with EVAL_QUOTA_STOP. A quota is never retried; --retries covers other errors only. Detection never relies on rateLimitType.
  • No API fallback. The API billing variables are removed before the helper, the preflight or any child runs.
  • Resume. Per-job JSONL records (0700 directory, 0600 files). A rerun skips ok jobs whose prompt and args hash is unchanged.
  • Batch helper gate for full runs. --scope full re-runs itself under claude-batch run --repo <repo> --class standard|fable -- …. That external helper pins one seat and trips a durable circuit on the first quota error (GSTACK_CLAUDE_BATCH overrides the command). Inside the batch, the driver checks the marker env (MERIT_CLAUDE_BATCH_ID, overridable with GSTACK_CLAUDE_BATCH_MARKER), then runs <helper> preflight before the first call.
    • A missing helper is refused with EVAL_BATCH_HELPER_MISSING (exit 3), and nothing falls back to unwrapped calls.
    • A failed preflight, or a helper that never sets the marker, is refused with EVAL_BATCH_PREFLIGHT_FAILED.
    • --scope smoke (at most 10 jobs, 2 workers) needs no helper.
  • API-funded eval routes are unchanged.
  • Result codes. The three new codes live in lib/result-codes-eval-run.ts, spread into RESULT_CODES the same way as result-codes-release.ts. This keeps lib/result-codes.ts under the module-size limit. Each code has a docs/troubleshooting.md anchor.

Tests

test/eval-cli-run.test.ts (20 cases) drives the real bin against a fake claude and a fake claude-batch on a minimal PATH. No real model call and no network. The fake claude keeps its own in-flight counter under a lock, so concurrency is observed outside the runner. Coverage:

  • the classifier on the real rejected-call record shape, null rateLimitType, stderr-only records, stream rate_limit_event, and answers that merely mention "rate limit"
  • a quota stop that is never retried, with nothing starting after it
  • exit 75 treated as fatal
  • the in-flight bound
  • the billing scrub at all three process layers
  • resume, including a changed-args rerun
  • file modes
  • the helper gate: missing helper, failed preflight, wrap-then-preflight order, and a helper that never sets the marker
  • smoke limits
  • template structure

Mutation checks:

  • removing the quota stop fails 5 tests
  • retrying a quota fails 2
  • removing the billing scrub fails 1
  • passing the unscrubbed env to the helper fails 1
  • the pre-review classifier and resume fail the 2 new review cases

Checks run

  • bun test test/eval-cli-run.test.ts: 20 pass, 0 fail.
  • Focused set (placeholders, troubleshooting anchors, skill validation, gen-skill-docs, pricing/eval-plan): 1207 pass, 0 fail.
  • bun run typecheck: clean.
  • bun run typecheck:test: no new diagnostics.
  • bun run gen:skill-docs regenerated eval-plan/SKILL.md and the agents digest.
  • The local bun run test did not pass on this machine: 434 failures on this branch, and 634 on base 952f95a. The failures are in PTY, browser, Docker, pregate and install tests. The files this change plausibly touches fail identically on HEAD and base when run alone (289 pass / 24 fail both). CI is the acceptance run.
  • An outside review (GPT-5.6 Sol, xhigh) found two P1s and three P2s. All are fixed in 684e56a with regression tests.

Follow-ups

  • ask: Once the external claude-batch helper is installed, run one --scope smoke and one tiny --scope full against it to confirm the live contract (marker, preflight, exit 75 propagation).

session: gstack-contrib-eval-quota-stop

merit-blake and others added 8 commits October 18, 2026 10:30
…uota stop

gstack-eval-plan cli-run runs jobs as `claude -p --output-format json`
children through a pull-based pool of --workers slots. The first quota or
rate-limit result (a structured signal, a failed call's quota text, or exit
75 from the batch helper's open circuit) stops the run: no new job starts,
unstarted jobs are recorded cancelled, and a quota is never retried. The
child env drops the API billing routes. Records go to cli-run.jsonl, so a
rerun resumes. A full run re-executes itself under the seat-pinning batch
helper (GSTACK_CLAUDE_BATCH, default claude-batch) and runs its preflight
inside; smoke scope runs at most 10 jobs on 2 workers without it.

New result codes EVAL_QUOTA_STOP (exit 75), EVAL_BATCH_HELPER_MISSING and
EVAL_BATCH_PREFLIGHT_FAILED live in lib/result-codes-eval-run.ts, spread
into RESULT_CODES so lib/result-codes.ts stays under the module-size
ratchet; the fixtures that copy that module's import closure list it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the run

The /eval-plan template now says a subscription (CLI-metered) run goes
through `gstack-eval-plan cli-run`, never a hand-written loop, and the
first quota or rate-limit error ends it (EVAL_QUOTA_STOP): no retry, no
seat switch, no API key, no fallback. Step 3 runs CLI-metered pilot
attempts with `cli-run --scope smoke` and settles them `unknown`; Step 4
names `cli-run --scope full --class <standard|fable>` for the approved
full run. The preregistration template's stop rules, docs/skills.md and
the placeholder allowlist (<jobs.jsonl>, <standard|fable>) follow, plus a
structure test for the template.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The helper and its preflight now see the same environment as the
children, with the API billing routes removed, so nothing downstream of
a quota stop can fall back to the API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…me binds prompt and args

Outside review (Sol) findings:
- A structured quota record on stderr, or a stream-json rate_limit_event
  with rate_limit_info.status=rejected, now stops the run at any exit code.
  Before, it classified as a retryable error.
- Resume skips an ok record only when the job's prompt and args hash
  matches, so a rerun with a changed model arm is not silently skipped.
- The output directory is 0700 and its files 0600, also on resume.
- Tests seed every API billing variable and check the helper too, and pin
  that API-funded pilot attempts keep the project's own command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… first quota error

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@trunk-io

trunk-io Bot commented Oct 18, 2026

Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@github-actions github-actions Bot changed the title feat(eval-plan): cli-run stops a subscription eval at the first quota error v1.91.138.0 feat(eval-plan): cli-run stops a subscription eval at the first quota error Oct 18, 2026
@capy-ai

capy-ai Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Thanks @merit-blake. The incident write-up is excellent, and the core of this is right: a bounded pull pool, a stop on the first quota signal that never relies on rateLimitType, removing the API billing variables before any child starts, private resumable JSONL, and the skill rule against retries, seat switches or API fallback. We'd like to land it. Before we can, a few changes:

  1. The full path depends on tooling gstack users don't have. --scope defaults to full, and full requires --class standard|fable and then re-runs itself under claude-batch with MERIT_CLAUDE_BATCH_ID (lib/eval-cli-run.ts:40,346-347, bin/gstack-eval-plan:278-309). For anyone else, an approved full run refuses with EVAL_BATCH_HELPER_MISSING (exit 3). Because the skill also says never to hand-roll a loop, that leaves no sanctioned way to run one. Please make the helper wrap opt-in (only when GSTACK_CLAUDE_BATCH is set or the marker is present) and run directly otherwise. Use a neutral marker name such as GSTACK_CLAUDE_BATCH_ID, make --class optional, and drop fable from the public surface (template, troubleshooting text and the <standard|fable> placeholder).
  2. Job args can turn API billing back on. Only -p, --print, --output-format and --input-format are reserved (lib/eval-cli-run.ts:35,186). A job with ["--settings", "{\"env\":{\"ANTHROPIC_API_KEY\":…}}"] passes through. Please reserve --settings and --setting-sources, and say in the docs that env / apiKeyHelper in the user's own ~/.claude/settings.json is out of scope.
  3. A "429" anywhere stops the run. \b429\b is matched over all of stdout and stderr on a failed call (:154-157), so {"is_error":true,"duration_ms":429} reads as a quota hit. Match it only in error text or the structured status fields.
  4. --out is chmodded even when it already exists (:236-239), so --out . would make the current directory 0700. Chmod only a directory you created, or refuse an existing one that isn't private.
  5. Drop the v1.91.138.0 stamp (VERSION, package.json, agents-digest, CHANGELOG heading). Main is past it, and those four files are the only merge conflicts. Put the entry under ## [Unreleased] and we'll stamp it when it lands.

Everything else checked out on a merge with current main: typecheck, fresh generated docs, the size guard, parity, and 599 focused tests including your 41. The fake-claude quota run also behaved: 2 calls, 3 jobs cancelled, exit 75, the key unset in the child, 0700/0600 output, and a correct resume. Once these are in, we'll port it to a crew branch with your authorship so the paid gate can run with secrets.

@capy-ai

capy-ai Bot commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

Thank you for this. A crew lane is picking it up to land on current main with Co-authored-by credit to you.

@capy-ai

capy-ai Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Thank you @merit-blake. The idea at the heart of this PR shipped in v1.92.50.0, through #3751 and wave #3788: a seat-metered run should stop at its first quota or rate-limit error, with no retry. It's now a stop rule in the /eval-plan skill and its preregistration template, credited to you in the changelog.

We wrote that rule as skill prose on current main and didn't merge this branch. The cli-run runner here depends on an external batch helper (claude-batch / MERIT_CLAUDE_BATCH_ID), strips API keys from the environment, and edits a workflow file. Those pieces are more than the stop rule needs, and they touch areas (secrets handling and CI workflows) we keep to changes we write and review ourselves. #3751 also adds pilot start --seat-metered, so subscription arms like claude -p can be declared, kept out of the spend cap, and approved.

Closing this in favor of #3751. Thanks again for the idea.

@capy-ai capy-ai Bot closed this Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant