Repository navigation
v1.92.2.0 evals: no-gstack arms for the model panel, arm-benchmark, safety and routing evals - #3708
Merged
Merged
Conversation
…no-gstack rows Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…cells, one campaign, results.md (GST-468) Each panel model also runs as <model>#no-gstack: the same fixture, ticket, tools, limits, route and reservation, with no skill folder and no routing line. The arm identity keys the ledger attempt, test name, row and summary; candidate folders are named after the fixture project and the seed commit has a neutral author. summarize reads one campaign (run id and PREREG.md sha256) and refuses a duplicate cell; suggestionLine reads the skill arms only, so the preregistered particle-1 line is unchanged. renderResults writes results.md, which passes gstack-eval-plan results --check against the new test/fixtures/model-panel/PREREG.md (supersedes GSTA-173 for later runs). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…, both-order paired judge, spend ledger (GST-469) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…an/no-gstack-arms-existing-evals
…470) GSTACK_SAFETY_ARM and GSTACK_ROUTER_ARM take a third value, none: the same fixture, tools and user request with no skill excerpt, rule text, step file or installed skill. Under none every behavioral check is recorded as an observation (none-arm-observations.jsonl) and never fails the case or writes contract-violations.jsonl; harness preconditions still assert. The four safety prompt builders (ship-evidence, codex-boundary, codex-consult, design-risk) and the router fixtures move into helpers; the rule, removed, current and before prompts stay byte-identical, proven against reference copies of main's code. pair-agent and ios-demo refuse none with the reason (NO_NONE_ARM). GSTACK_ROUTER_ARM now throws on an unknown value instead of silently running current. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…an/no-gstack-arms-existing-evals
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…/6, $7.22) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…ms-existing-evals
…ctable by --case The lanes moved each cell's testName into helpers, so the static detector no longer saw a computed registration and B4 failed for all eight cases. Same runtime test names; the paid runs are unaffected. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
|
😎 This pull request was merged. |
…for the no-gstack arms scripts/judge-calibration.ts is now an arm-benchmark paid touchfile, so it can no longer be a free consumer of test/fixtures/judge-calibration/** (CI free-suite 8). The eval-plan Step 5 fence names <prereg-path> and <results-path>; allowlist both as PATH placeholders (CI free-suite 19). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…utput and the results usage error Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
… corpus Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This was referenced Oct 31, 2026
garrytan
added a commit
that referenced
this pull request
Oct 31, 2026
Resolves bin/gstack-eval-plan help and docs/skills.md by keeping both sides (#3708's results --check and this PR's per-arm report and --seat-metered text). Raises the eval-plan eager context ceiling 4887 -> 5021 (+134 tok): #3708 and this PR each fit alone; together eval-plan/SKILL.md measures 5021. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This was referenced Oct 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Garry's no-gstack rule (R44: "all of our head to head tests have to test no gstack from here ... Do it for science") now reaches the head-to-head evals that existed before #3690.
<model>#no-gstackwith no skill installed. The paid test asserts the no-gstack arm never consults a skill. New preregistration at test/fixtures/model-panel/PREREG.md, and the run writes a results.md table.arm-pairconfig in scripts/judge-calibration.ts). The old per-diff judge stays beside it as a diagnostic. New PREREG at test/fixtures/arm-benchmark/PREREG.md.nonearm that sends the same scenario with no skill text (test/helpers/none-arm.ts and four task builders). It records observations and never counts as a contract violation. pair-agent and ios-demo refusenoneand say why.gstack-eval-plan results --check <prereg> <results.md>fails withEVAL_REPORT_NO_GSTACK_ROWwhen a results table is missing a no-gstack row for any arm the PREREG lists. /eval-plan Step 5 now requires it before any number is published.--caseselector can see again.results --check now refuses a results file that drops a no-gstack row (before: the command did not exist). Image: render failed (PR_SHOTS_RENDER_FAILED), so the text version follows.
Before/after as text: results --check now refuses a results file that drops a no-gstack row (before: the command did not exist)
Paid validation (lane cap $35, written by GStack Minder)
results --check: ok 4/4 no-gstack rowsresults --checkok on bothTotal $15.36 settled, $15.96 worst case. An earlier panel launch (ng5-panel-20261031T163926Z) died when the machine slept, with 44 of 48 cells done. It was re-run once as a new campaign under the same ledger, and its cells are not in the numbers above. Both instruments are near ceiling, so these rows show the no-gstack arm works end to end. They don't measure skill value.
Why
"All of our head to head tests have to test no gstack from here ... Do it for science." The GST-461 audit found five head-to-head evals with no no-gstack arm. This PR adds one to four of them, and a results file can't be published until each no-gstack row sits beside its variants. /benchmark-models (GST-472) gets its own plan.
Blast radius
Test harnesses, fixtures, one bin subcommand, one result code and docs. The new result code makes PR selection pick every gate case, the same as #3690 (selection read in eval-select). The safety and router
nonearms are set only by hand and get no paid run here; their first comparison gets its own preregistration.Approvals
Source: gstack-approvals record for garrytan/no-gstack-arms-existing-evals (display only; not an approval citation).
Summary
Garry asked that every head-to-head eval also test no gstack. The model panel, arm-benchmark, safety and failure-routing evals now carry a no-gstack (or
none) arm, arm-benchmark gets a calibrated paired judge, andgstack-eval-plan results --checkkeeps a results file from publishing until every arm has a filled row. This /ship run also fixed two CI failures at ea50880: the free-fixtures consumer list and the placeholder allowlist for<prereg-path>and<results-path>.Test Coverage
Coverage gate: PASS (95%). Tests: 3 new in test/eval-plan-results.test.ts (Step 7 coverage audit), all 11 tests in that file green.
Test value: 3 tests written, 0 rejected by the authoring gate, 1 existing test extended, 1 path weakly covered (the aligned-separator half of the first new test cannot fail; see Review notes).
Tests ran: focused 790
Tests ran: typecheck count unavailable
Pre-Landing Review
Pass 1: 9 findings, 3 blocking. Two were fixed (FREE_FIXTURES consumer list, placeholder allowlist). The third is a gate finding kept under approval a-f282a0395e2b (below). Pass 2 (specialist batched, testing/security/state-machines): no findings.
REVIEW_LOOP: passes=2 blocking_found=3 deferred=0 noted=6 fixed=2 scope=frozen dispatches=7
Gate edits: partial (50 of 61 eligible read), 1 finding — 1 kept (reasons below), 0 restored, 0 open
prereg-with-results, test/helpers/arm-pair-judge.ts. Reason: plan r2 decision 5 puts the paid validation in this PR (approval a-f282a0395e2b, GStack Minder gate_rev 2). The PREREGs were committed at 3dcb1ac (16:19Z) and c323d4f (16:24Z), before the first paid run at 16:36Z (32cb224), and the model-panel PREREG sha 998f9554… matches its results.md.Outside review (openai/gpt-6.1-sol, Capy subagent): completed, 5 findings marked Blocks: yes; judged below the floor by named rulings (Review notes). Native adversarial (Claude): 5 findings, all Low. Targeted gate reader: 3 informational, all duplicates.
Review notes
Below the blocking floor; recorded here and not fixed in this PR.
results --checkreturns ok forstack: nonebefore it checks that the preregistration passesprereg --check(lib/eval-plan-report.ts:34). Four readers raised it. Ruling: own-artifact hardening. docs/troubleshooting.md says the code also fires on a failing preregistration, which is not true on this path.results --checkpasses it. Ruling: own-artifact hardening.nonerecords a pass (test/helpers/none-arm.ts:43, test/skill-e2e-failure-routing.test.ts:60). Ruling: unwired path; only a person sets it, and CI never does.Exploratory QA
Probes 001–006 all green (results --check full and missing, prereg --check on both PREREGs, results --check on both paid results.md). Verdict: pass, no open items.
Design Review
No frontend files changed — design review skipped.
Eval Results
Paid runs are listed under Paid validation above. No new paid eval ran in this /ship; a paid eval of the changed /eval-plan prompt is deferred to gstack/full-gate.
Greptile Review
Greptile triage: UNAVAILABLE (not configured for this repository).
Plan Completion
24 items: 23 done, 1 changed (the arm-pair calibration corpus).
Verification Results
Step 8.1 obligations: all six met; none failed, blocked or not run.
TODOS
No TODO items completed in this PR.
Documentation
Status: updated. One factual addition to docs/skills.md. No blockers. One documentation debt item under test/ (read-only for this ship).
Scope: base 5ba425a to head 9c6f9e7 on garrytan/no-gstack-arms-existing-evals: 44 committed files, with nothing staged, unstaged or untracked. I checked the changed docs (docs/TESTING_INTERNALS.md, docs/troubleshooting.md, eval-plan/SKILL.md.tmpl, the changelog fragment and both PREREG.md files) against lib/eval-plan-report.ts, bin/gstack-eval-plan, lib/result-codes-eval.ts and the test helpers they describe. I also searched the other docs for references to the arm-benchmark, the model panel, the safety and router arms and eval-plan.
Documentation health
results --checkCoverage debt:
gstack-eval-plan results --checknow has reference docs (troubleshooting, skills.md, CLI help), a how-to (eval-plan Step 5) and an explanation (TESTING_INTERNALS). The arm-benchmark and model-panel no-gstack arms are explained in TESTING_INTERNALS and specified in their PREREG.md files. They have no how-to for running a paid comparison, but each PREREG names its command. test/fixtures/judge-calibration/README.md still says every corpus holds 24 items and that scripts/judge-calibration-configs.ts defines every configuration. The arm-pair corpus has 16 paired items, its configuration is in test/helpers/arm-pair-judge.ts, and its comparison-1 result (16/16 agreement, 0 flips, $7.22, lands) is missing from the results table.Diagram drift: none. No architecture diagram covers the changed files.
Test plan
bun teston 35 files — 790 pass, 0 fail (Step 16 at b32a013)bun run typecheck— passbun run gen:skill-docs— no diffPREGATE_VERIFY: ok)