Repository navigation
Conversation
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…d CLI wiring Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…mmed band (arms share rows) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…ut, price row still required, a real charge settles as spent) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…d arm Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…s no $N token Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…a charge Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
E2E Evals: ✅ PASS0 executed, 0 reused rule/judge records | 0 manual accepted (unscored; no score-cache credit) | $0 rule/judge cost | reconcile exit: 0 Profile: pr / mode Rule and judge shards
Fail-closed reconciliationSliced lane: planner → duration-packed executors → fail-closed report. Behavior cases run a pre-registered 3-trial panel (PASS at 2/3 with no contract violation); a PASS 2/3 is shown with its failed trial, never as a clean pass. Reused results retain their original provenance and expiry. Dispatched narrow screen (gstack-ship-queue ready --screen) of merge commit |
Resolves bin/gstack-eval-plan help and docs/skills.md by keeping both sides (#3708's results --check and this PR's per-arm report and --seat-metered text). Raises the eval-plan eager context ceiling 4887 -> 5021 (+134 tok): #3708 and this PR each fit alone; together eval-plan/SKILL.md measures 5021. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
|
Frozen in wave |
* fix(eval-plan): per-arm pilot projection and --seat-metered arms (#3653) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(autoplan): guard tolerates Claude Code's cwd-reset footer and recovers after an unreadable init (#3651) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes (journal reader strips the cwd-reset footer; an init with no text block no longer locks the session) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: journal reader skips a failed or unreadable init, as invocation() does Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(autoplan): --ceo-only runs the CEO review alone with zero questions (GST-537) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(eval-plan): all-seat pilot, single-attempt arm and --seat-metered CLI wiring Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(eval-plan): summed deviation adds arm deviations, matching the summed band (arms share rows) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes (test pins an unpriced seat-metered arm) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes (seat-metered declaration fixed per --out, price row still required, a real charge settles as spent) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: classify autoplan-ceo-only in the Codex host metadata Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: a seat-metered arm that settles a charge is projected as a priced arm Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: select autoplan-ceo-only on the skills it runs; say it needs Claude Code Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(skills): /eval-plan --seat-metered and per-arm projection Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(eval-plan): skill prose stays under its context budget and carries no $N token Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: run autoplan-ceo-only in the marathon lane Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(skills): a seat-metered arm leaves the seat set once it settles a charge Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: changelog entry for /autoplan --ceo-only Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(autoplan): init_failed covers unreadable init output and the cwd-reset line (#3651) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: count the autoplan-ceo-only paid file in the shard inventory Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(autoplan): #3651 tests pass the test-code type ratchet Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(dx): CAP finding titles carry a one-line description (GBRA-223 claim 2) The DX error contract told both autoplan DX voices to title capped findings exactly CAP4:<path>:<item>, so gates, tasks and reconcile saw a bare code. The title is now CAP4:<path>:<item> <what fails>; the grader already parses that shape. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(autoplan): reconcile pairs same-file findings that carry no line (GBRA-223 claim 1) Plan reviews name files, rarely lines, and reconcile associated by location only when both findings had a line, so findings both voices raised on one file were filed new and native-only. A same-file pair with a missing line is now associated as unverified (basis file), never confirmed; a CAP code no longer counts in title comparison. DRIVER.md names submit --reconcile. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(autoplan): open Critical and High findings reach the gate as apply items (GBRA-223 claim 3) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(autoplan): a spend cap binds on every unattended host; --estimate-usd persists; DRIVER names --usage-usd (GBRA-223 claim 4) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(autoplan): run binds complete api results, names a truncated response, --max-tokens; DRIVER documents run (GBRA-223 claim 6) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(autoplan): carry an item-by-item answer when the later gate asks nothing new (GBRA-223 claim 7) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(dx): trim the compact error contract so plan-devex-review stays under its size pin Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes (p1b applies finding fixes, api attempts reserve no host default, cross-file title pairing, paired severity counted) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: review fixes (priced api reservations, no file-only twins, two-pass pairing, partial replies keep p1-dependent defaults open, outside refusal degrades) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * P7: record whose words answered a gate (--words-by agent:<id>), never carry an agent's own words, import each row under its own answerer (GEVA-2) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: review pass 2 (p1b needs --rec-hash after recommend; api calls send the output limit they reserve for) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * changelog: fragment for the /autoplan field-report wave (GST-672) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: captain record notes --words-by agent attribution Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: follow finding apply items in approval-class and Captain flow tests; bind run-api limit to DEFAULT_MAX_TOKENS Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * wave autoplan: lower the gemini plan-devex-review pin to its exact size after #3781's DX trim Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(autoplan): CAP titles on different paths never match by title in reconcile or confirm rounds (wave review) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(autoplan): a truncated api call settles its cost and a retry is admitted afresh against the cap Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(autoplan): an applied finding stays open until the revised plan is reviewed; a shared location alone never twins two findings Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(approval): a plan gate answered in an agent's own words is not owner approval Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(gate): a new answer replaces the earlier answer's authorship and words Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(autoplan): a failed api call with no usage keeps its attempt open; reservations price the bound prompt (wave fix-check) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(autoplan): a spend cap fails closed: only a call refused before sending keeps its attempt; a sent call with unknown usage settles at its reservation Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(autoplan): the 16384 output default applies to both providers; changelog for the wave's integrator fixes Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(autoplan): how run settles a failed api call; changelog scopes it to the native voice Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(autoplan): a spend cap fails closed on every exit of run; an unpriced model reserves the default once a cap is set Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(autoplan): every sent call settles; an unpriced model under a cap reserves the default Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(autoplan): DRIVER and help say a cap reserves in any session kind and how run settles a sent call Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
What changed
The same pilot (2 attempts each on gpt-5.4-mini at $0.02 and on gpt-6-astra settled
unknown, 100 items, $5 cap) reported on main and on this branch. Main can never approve it and projects $4.00; this branch projects the priced arm at $2.00, lists the seat arm on its own line and says it fits.Before/after as text
Why
#3653: a paid eval that compares a priced API model with
claude -pon a subscription could never be approved, because the subscription arm settlesunknownandpilot reportrefused any pilot with unknown charges. The same report also pooled every arm into one mean, so a priced arm's cost was multiplied across arms that cost nothing, and two arms at different prices got a band wide enough to fail a pilot that fit the cap. Now a subscription arm is declared up front withpilot start --seat-metered <arm>, and each priced arm is projected from its own attempts.Blast radius
Only
gstack-eval-plan pilot startandpilot reportand the /eval-plan skill prose change; the spend ledger (lib/spend-ledger.ts) and the dry run are untouched, and a pilot started without--seat-meteredkeeps every arm priced, so existing pilots only see the per-arm projection (script ran: the 640-test eval-plan lane, plus the base/head reports above). A seat arm still needs alib/pricing.tsrow, and an attempt on it that settles a real charge counts as spent and puts the arm back in the projection, so the cap can't be dodged by declaring an arm seat-metered (walked failure path and pinned in test).Approvals
No approvals recorded for branch gst-566-eval-plan-seat-metered.
Summary
"A seat-metered arm makes every pilot unapprovable" was the whole of #3653's first half; its second half was the pooled mean. The fix projects each arm on its own and sums them, adds
--seat-meteredso an arm with no per-call charge holds none of the cap and gets its ownPILOT_SEAT_METERED:line, and fixes the declaration per--outso a restart can't quietly change it. The skill also tells the operator to stop a seat-metered run at its first quota or rate-limit error, with no retry. That stop rule is the idea from #3273 by @merit-blake, rewritten here as skill prose; none of #3273's code was carried over (its batch runner, environment key handling and workflow edit were declined).Security read: nothing here makes network calls, reads environment variables or secrets, widens permissions, edits workflows or install scripts, or adds a dependency.
Test Coverage
7 new tests in
test/pricing-eval-plan.test.ts, all in the existing priced-pilot suite:unknown--outpilot start --seat-meteredreaches the ledger through the CLITests: 1571 test files in the repo, 0 new files; 7 tests added to an existing file.
Regression proof — fails at HEAD's base code: 6 of 7 fail against main 3bfe566's
lib/eval-plan.ts(26 pass, 6 fail with this branch's test file copied onto main); the price-row guard passes at main by design · passes after fix: yes (32 pass, 0 fail).Tests ran: evalplan 640
Tests ran: guards 13
Pre-Landing Review
Review found and fixed two cap bypasses before this PR opened; nothing blocking is left.
pilot startin the same--outwithout--seat-meteredreused the $0 reservations, so priced calls got past the cap. Fixed in 8c9b61c: a pilot's seat set is fixed when it starts, and a change needs a new--out. The same commit fixes its Medium (a seat arm still needs a price row, because the dry run requires one) and the adversarial Medium (a real charge on a seat attempt could not be recorded; it now counts as spent).bin/gstack-eval-planhelp line anddocs/skills.mdparagraph; both sides are kept. Merged, eval-plan/SKILL.md measures 5021 tokens against its 4887 eager ceiling: each PR fit alone, and v1.92.2.0 evals: no-gstack arms for the model panel, arm-benchmark, safety and routing evals #3708 used the headroom. The ceiling goes to 5021 (+134) intest/fixtures/context-budget.json, and only that one entry changes. A fix-check on the resolution hunks found nothing (tree 1037735).projected_usdin the JSON report is a subtotal when an arm has no numeric settles. eval-plan: seat-metered pilot arms remain unapprovable; the mixed-arm projection extrapolates known costs across unknown arms #3653 asks for that subtotal, the text labels itknown_arms=k/m, and the verdict staysunknown.REVIEW_LOOP: passes=2 blocking_found=0 deferred=0 noted=2 fixed=1 scope=frozen fix_checks=4 dispatches=2
Gate edits: one test file extended (
test/pricing-eval-plan.test.ts, additions only); no test, threshold, budget or CI gate relaxed. The first remote pregate caught a context-budget overrun, fixed in 0dce80e by trimming the skill prose, not by raising the ceiling.Outside review: the cross-family outside voice ran in passes 1 and 2; its High and Medium are fixed above.
Review notes
Non-blocking findings, listed and not looped (crew rule 23 floor):
--seat-metered <arm>for an arm that is already declared, when that seat arm also settled a real charge. Display only.PILOT:line'sunknown=excludes seat arms, while the spend table's "unknown charges" includes them (visible above:unknown=0beside2). Display only.pilot.jsonoutside the ledger lock, so two concurrent starts in one--out, or a start that failed partway through reserving, could mix seat sets. Both need an unusual earlier event.exceedsfor a run that fits). That is the safe direction for a cap.Exploratory QA
gstack-eval-plan pilot start / settle / reportbun --no-env-file --config=/dev/null bin/gstack-eval-plan~/.capy/work/shipA/qa-probe/fx*(owned, synthetic dry run; no provider calls)Contract outcomes (re-run at 2321dc5)
pilot start --arms gpt-5.4-mini,gpt-6-astra --n 2 --seat-metered gpt-6-astraestimate=$0.0000×2,status=reservedpilot settle --attempt pilot-gpt-6-astra-1 --usd 0.5SETTLED … spent=$0.5000; the report then projects gpt-6-astra as priced--seat-meteredmust name a listed armpilot start --arms gpt-5.4-mini --seat-metered gpt-6-astra--outdir--outpilot starton the same--outwithout--seat-meteredpilot report --total 100PILOT_ARM, onePILOT_SEAT_METERED, verdict within → as shown aboveDiscoveries
Coverage limits
No live provider or
claude -pcall ran; costs are synthetic settles. No new tests came from QA; the behaviors above are each pinned by a unit test listed under Test Coverage.Design Review
No frontend files changed — design review skipped.
Eval Results
No paid eval ran in this ship. The skill prose change was checked by the free gates: gen-skill-docs freshness, context-budget ratchet, parity suite and the skill size guard. The paid screen runs at
gstack-ship-queue ready --screen.Plan Completion
Plan: GSTA-218 (approved). #3653's per-arm projection and
--seat-metered(D1) and #3273's stop rule as prose only (D2) are done here; #3651 (D3) ships as its own PR.Issues
Fixes #3653
Verification Results
The plan's checks: failing tests first (6 fail at main, pass here), the #3653 repro (
~/.capy/work/repro3653, now approvable), and the QA probes above: all PASS. Remote pregate on Ubicloud: 18185 tests, 0 failed, at tree 1037735 (merge commit 1514240). Two remote-only lanes (macos-named-regressions, windows-free-shard) clear from this PR's Actions runs.TODOS
No TODO items completed in this PR.
Documentation
Status: updated. One factual edit to
docs/skills.md: a seat-metered arm stays out of the projection only until an attempt settles a real charge, matchinglib/eval-plan.ts.Scope: base 3bfe566, head 0dce80e, no staged/unstaged/untracked changes; committed changes in
bin/gstack-eval-plan,lib/eval-plan.ts,eval-plan/SKILL.md(.tmpl),docs/skills.md,changelog.dentry and the test file. Docs touching eval-plan or the pilot subcommands were checked.Documentation health:
Coverage debt: the
EVAL_BUDGET_EXHAUSTEDentry in docs/troubleshooting.md omits--seat-metered. Diagram drift: none found; Step 1.5 diagram check not run.Test plan
bun test test/pricing-eval-plan.test.ts test/eval-plan-no-gstack-arm.test.ts test/eval-plan-prereg-diagnostics.test.ts test/eval-models.test.ts test/catalog-budget.test.ts test/gstack-skill-start.test.ts test/auq-error-fallback-hook.test.ts test/context-budget-ratchet.test.ts test/gen-skill-docs.test.ts: 640 pass, 0 fail at 1514240bun test test/parity-suite.test.ts test/module-size-ratchet.test.ts test/skill-md-size-guard.test.ts test/spawnsync-timeout-tripwire.test.ts: 13 pass, 0 fail at 1514240UBI_OWNER=gsta218 gstack-pregate --stage tests --remote --spend-cap 2 --timeout 2700 --allow-repo-commands --base main: 18185 tests, 0 failed (tree 1037735)🤖 Generated with Claude Code