Skip to content

v1.92.2.0 evals: no-gstack arms for the model panel, arm-benchmark, safety and routing evals - #3708

Merged
capy-ai[bot] merged 18 commits into
mainfrom
garrytan/no-gstack-arms-existing-evals
Oct 31, 2026
Merged

capy-ai[bot] merged 18 commits into
mainfrom
garrytan/no-gstack-arms-existing-evals

Conversation

@garrytan

@garrytan garrytan commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner
{
  "schema_version": 1,
  "pr": 3708,
  "head": "b32a01331e5fa4a2fbf219c0ed545eccb4935331",
  "version": "1.91.376.0",
  "base": "main",
  "gate": {
    "focused": {
      "pass": 790,
      "fail": 0,
      "skip": 0
    }
  },
  "session_kind": "unattended",
  "head_tree": "2e2052f3f5d96deb705824f487505fb6f595e42b",
  "product_patch_id": "5f014cad38a51946a154f18a83a0a90a21d975ad",
  "gstack_version": "1.91.376.0",
  "review_depth": {
    "depth": "standard",
    "class": "code",
    "runtime_lines": 100,
    "rules": "150435b0a543019a",
    "base": "5ba425af4b9a6734317d43b6ea9b4077d08ff1af",
    "registry": "5ba425af4b9a6734317d43b6ea9b4077d08ff1af",
    "gate_scan": {
      "tagged": 0,
      "head": "b32a01331e5fa4a2fbf219c0ed545eccb4935331"
    },
    "reasons": [
      "runtime=100>50"
    ]
  },
  "pregate": "requires-remote cleared @2e2052f3f5d9 (1 checks, current for b32a01331e5f)",
  "requires_remote": [
    "free-tests.yml/cso-macos-launcher: https://github.com/garrytan/gstack/actions/runs/41227959259",
    "free-tests.yml/macos-named-regressions: https://github.com/garrytan/gstack/actions/runs/41227959259",
    "windows-free-tests.yml/windows-free-shard: https://github.com/garrytan/gstack/actions/runs/41227959245",
    "pregate.json/remote_gate: remote-gate tree=2e2052f3f5d9 tests=24433"
  ],
  "issues": {
    "fixes": [],
    "refs": [],
    "flakes": []
  },
  "docs": {
    "status": "updated",
    "scope": "docs/, README.md, CONTRIBUTING.md, AGENTS.md, CLAUDE.md, eval-plan/SKILL.md.tmpl; test/ read-only (fixture README debt noted)"
  },
  "review_loop": {
    "passes": 2,
    "blocking_found": 3,
    "deferred": 0,
    "noted": 6,
    "fixed": 2,
    "scope": "frozen",
    "dispatches": {
      "specialist": 6,
      "red-team": 1
    }
  }
}

What changed

Garry's no-gstack rule (R44: "all of our head to head tests have to test no gstack from here ... Do it for science") now reaches the head-to-head evals that existed before #3690.

  • Model panel (GST-468). Every model runs twice, with gstack's skill and as <model>#no-gstack with no skill installed. The paid test asserts the no-gstack arm never consults a skill. New preregistration at test/fixtures/model-panel/PREREG.md, and the run writes a results.md table.
  • arm-benchmark (GST-469). Three with/without pairs per task under a $25 spend ledger, a pinned model, project-named folders and a neutral git identity. A new paired judge (test/helpers/arm-pair-judge.ts) scores both diffs in both orders and only decides after it meets its calibration bar (new arm-pair config in scripts/judge-calibration.ts). The old per-diff judge stays beside it as a diagnostic. New PREREG at test/fixtures/arm-benchmark/PREREG.md.
  • Safety and failure-routing evals (GST-470). A none arm that sends the same scenario with no skill text (test/helpers/none-arm.ts and four task builders). It records observations and never counts as a contract violation. pair-agent and ios-demo refuse none and say why.
  • Results check (GST-471). gstack-eval-plan results --check <prereg> <results.md> fails with EVAL_REPORT_NO_GSTACK_ROW when a results table is missing a no-gstack row for any arm the PREREG lists. /eval-plan Step 5 now requires it before any number is published.
  • paid-shards: both paid test files name their cell tests in a form the --case selector can see again.

results --check now refuses a results file that drops a no-gstack row (before: the command did not exist). Image: render failed (PR_SHOTS_RENDER_FAILED), so the text version follows.

Before/after as text: results --check now refuses a results file that drops a no-gstack row (before: the command did not exist)
--- Before · before2.txt · captured output
+++ After · after.txt · captured output
 $ gstack-eval-plan results --check PREREG.md results.md
+RESULTS: incomplete no_gstack_rows=3/4 rows=7/8 file=results-missing.md
+gstack-eval-plan results: the results file has no single table with a filled row for every arm the preregistration lists, so a no-gstack row is missing or not beside its variants (no filled row for claude-sonnet-5-5#no-gstack in the table that holds the other arms); fix: put one row per arm identity (each <model>#no-gstack beside its variants) in one markdown table of the results file, then rerun `gstack-eval-plan results --check`; publish no number before it says ok (EVAL_REPORT_NO_GSTACK_ROW)
+exit 1
-gstack-eval-plan: unknown command results
-  (usage text trimmed)
-exit 2
 

Paid validation (lane cap $35, written by GStack Minder)

Run Result Spend
arm-pair judge calibration (committed in 32cb224) bar met: 6/6 crossing pairs decided correctly in both orders, 0 position bias, 144 calls $7.22 of $8
Model panel, campaign 2026-10-31-1652 (eval-bg logs ng5-panel2-20261031T165253Z, ng5-panel3-20261031T165724Z), 120 cells Opus 5.5 15/15, no-gstack 14/15 · Sonnet 5.5 15/15 and 15/15 · particle-1 13/15, no-gstack 15/15 · DeepSeek V4.1 Flash 15/15 and 15/15. results --check: ok 4/4 no-gstack rows $5.37 settled + $0.60 held at reservation (cap $15)
arm-benchmark pilot (run 2026-10-31-1710, crud-endpoint) then full run (2026-10-31-1712, the other two tasks), Opus 5.5, paired judge deciding with build-discipline: checks 6/6, paired W/T/L 1/8/0 · no-gstack: checks 6/6, 0/8/1. results --check ok on both $2.76 (pilot projection $2.55 against a $21.80 limit)

Total $15.36 settled, $15.96 worst case. An earlier panel launch (ng5-panel-20261031T163926Z) died when the machine slept, with 44 of 48 cells done. It was re-run once as a new campaign under the same ledger, and its cells are not in the numbers above. Both instruments are near ceiling, so these rows show the no-gstack arm works end to end. They don't measure skill value.

Why

"All of our head to head tests have to test no gstack from here ... Do it for science." The GST-461 audit found five head-to-head evals with no no-gstack arm. This PR adds one to four of them, and a results file can't be published until each no-gstack row sits beside its variants. /benchmark-models (GST-472) gets its own plan.

Blast radius

Test harnesses, fixtures, one bin subcommand, one result code and docs. The new result code makes PR selection pick every gate case, the same as #3690 (selection read in eval-select). The safety and router none arms are set only by hand and get no paid run here; their first comparison gets its own preregistration.

Approvals

  • Plan r2 for GST-468–471, approved by GStack Minder at gate_rev 2 (autoplan run ng5-autoplan-20261031155208), raising this lane's cap to $35 in writing.
Date Asked Exact words Outcome Covers Still pending
2026-10-31 Plan r2 decision 5: run the paid validation in this PR (PREREGs frozen and committed first, then calibration, panel, arm-benchmark pilot and full run), under a 35 USD lane cap “rec, which is plan r2 with every recommendation. Your lane cap goes to $35 in writing: the panel cap stays $15, and arm-benchmark runs its pilot first and stops unless the projection fits $25.” (GStack Minder (Captain, GSTA-36), implementing Garry's R44 rule) approved results and the calibration bar landing in the same PR as their preregistration; PREREGs committed at c323d4f and 3dcb1ac before the first paid run (32cb224, 16:36Z) none

Source: gstack-approvals record for garrytan/no-gstack-arms-existing-evals (display only; not an approval citation).

Summary

Garry asked that every head-to-head eval also test no gstack. The model panel, arm-benchmark, safety and failure-routing evals now carry a no-gstack (or none) arm, arm-benchmark gets a calibrated paired judge, and gstack-eval-plan results --check keeps a results file from publishing until every arm has a filled row. This /ship run also fixed two CI failures at ea50880: the free-fixtures consumer list and the placeholder allowlist for <prereg-path> and <results-path>.

Test Coverage

Coverage gate: PASS (95%). Tests: 3 new in test/eval-plan-results.test.ts (Step 7 coverage audit), all 11 tests in that file green.
Test value: 3 tests written, 0 rejected by the authoring gate, 1 existing test extended, 1 path weakly covered (the aligned-separator half of the first new test cannot fail; see Review notes).

Tests ran: focused 790
Tests ran: typecheck count unavailable

Pre-Landing Review

Pass 1: 9 findings, 3 blocking. Two were fixed (FREE_FIXTURES consumer list, placeholder allowlist). The third is a gate finding kept under approval a-f282a0395e2b (below). Pass 2 (specialist batched, testing/security/state-machines): no findings.

REVIEW_LOOP: passes=2 blocking_found=3 deferred=0 noted=6 fixed=2 scope=frozen dispatches=7

Gate edits: partial (50 of 61 eligible read), 1 finding — 1 kept (reasons below), 0 restored, 0 open

  • kept: prereg-with-results, test/helpers/arm-pair-judge.ts. Reason: plan r2 decision 5 puts the paid validation in this PR (approval a-f282a0395e2b, GStack Minder gate_rev 2). The PREREGs were committed at 3dcb1ac (16:19Z) and c323d4f (16:24Z), before the first paid run at 16:36Z (32cb224), and the model-panel PREREG sha 998f9554… matches its results.md.

Outside review (openai/gpt-6.1-sol, Capy subagent): completed, 5 findings marked Blocks: yes; judged below the floor by named rulings (Review notes). Native adversarial (Claude): 5 findings, all Low. Targeted gate reader: 3 informational, all duplicates.

Review notes

Below the blocking floor; recorded here and not fixed in this PR.

  • results --check returns ok for stack: none before it checks that the preregistration passes prereg --check (lib/eval-plan-report.ts:34). Four readers raised it. Ruling: own-artifact hardening. docs/troubleshooting.md says the code also fires on a failing preregistration, which is not true on this path.
  • A table inside a code fence counts as a results table, one complete table is enough even if another table drops the row, and any text fills a cell (lib/eval-plan-report.ts:21-27). Ruling: own-artifact hardening.
  • renderArmResults writes 0 cells instead of dashes for an arm with no valid cells (test/helpers/arm-benchmark-harness.ts:360), so results --check passes it. Ruling: own-artifact hardening.
  • A stray GSTACK_SAFETY_ARM or GSTACK_ROUTER_ARM=none records passed:true under the gate case names with no arm field, and a router launch failure under none records a pass (test/helpers/none-arm.ts:43, test/skill-e2e-failure-routing.test.ts:60). Ruling: unwired path; only a person sets it, and CI never does.
  • The arm-benchmark per-task timeout (2,460 s) is above the default paid-shard supervisor (1,800 s). Ruling: conservative refusal; a kill is an INFRA red and a rerun. The full run took 209.42 s for all three tasks (ng5-armfull-20261031T171223Z).
  • A budget stop drops a sibling cell already run from the results table (test/skill-e2e-arm-benchmark.test.ts:209). A stopped run is incomplete under the preregistration's stop rule, and its charges stay in the ledger.
  • scripts/judge-calibration.ts imports a test helper, and the arm-pair config has no manifest price; the per-run cap in arm-pair-judge binds.
  • The aligned-separator half of the first Step 7 test cannot fail (test/eval-plan-results.test.ts:68).

Exploratory QA

Probes 001–006 all green (results --check full and missing, prereg --check on both PREREGs, results --check on both paid results.md). Verdict: pass, no open items.

Design Review

No frontend files changed — design review skipped.

Eval Results

Paid runs are listed under Paid validation above. No new paid eval ran in this /ship; a paid eval of the changed /eval-plan prompt is deferred to gstack/full-gate.

Greptile Review

Greptile triage: UNAVAILABLE (not configured for this repository).

Plan Completion

24 items: 23 done, 1 changed (the arm-pair calibration corpus).

Verification Results

Step 8.1 obligations: all six met; none failed, blocked or not run.

TODOS

No TODO items completed in this PR.

Documentation

Status: updated. One factual addition to docs/skills.md. No blockers. One documentation debt item under test/ (read-only for this ship).

Scope: base 5ba425a to head 9c6f9e7 on garrytan/no-gstack-arms-existing-evals: 44 committed files, with nothing staged, unstaged or untracked. I checked the changed docs (docs/TESTING_INTERNALS.md, docs/troubleshooting.md, eval-plan/SKILL.md.tmpl, the changelog fragment and both PREREG.md files) against lib/eval-plan-report.ts, bin/gstack-eval-plan, lib/result-codes-eval.ts and the test helpers they describe. I also searched the other docs for references to the arm-benchmark, the model panel, the safety and router arms and eval-plan.

Documentation health

File Status
docs/TESTING_INTERNALS.md current: every no-gstack arm claim checked against the source
docs/troubleshooting.md current: EVAL_REPORT_NO_GSTACK_ROW matches the code and its result-code entry
eval-plan/SKILL.md.tmpl current: Step 5 matches the CLI's output and exit codes
docs/skills.md updated: the /eval-plan section now covers results --check
test/fixtures/judge-calibration/README.md debt: does not describe the arm-pair corpus (read-only)
README.md, AGENTS.md current

Coverage debt: gstack-eval-plan results --check now has reference docs (troubleshooting, skills.md, CLI help), a how-to (eval-plan Step 5) and an explanation (TESTING_INTERNALS). The arm-benchmark and model-panel no-gstack arms are explained in TESTING_INTERNALS and specified in their PREREG.md files. They have no how-to for running a paid comparison, but each PREREG names its command. test/fixtures/judge-calibration/README.md still says every corpus holds 24 items and that scripts/judge-calibration-configs.ts defines every configuration. The arm-pair corpus has 16 paired items, its configuration is in test/helpers/arm-pair-judge.ts, and its comparison-1 result (16/16 agreement, 0 flips, $7.22, lands) is missing from the results table.

Diagram drift: none. No architecture diagram covers the changed files.

Test plan

  • focused: bun test on 35 files — 790 pass, 0 fail (Step 16 at b32a013)
  • typecheck: bun run typecheck — pass
  • bun run gen:skill-docs — no diff
  • remote pregate at b32a013 (tree 2e2052f3f5d9): 24433 tests, 0 failed, $0.20; CI obligations cleared from runs 41227959259 and 41227959245 (PREGATE_VERIFY: ok)

garrytan and others added 10 commits October 31, 2026 16:09
…no-gstack rows

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…cells, one campaign, results.md (GST-468)

Each panel model also runs as <model>#no-gstack: the same fixture, ticket,
tools, limits, route and reservation, with no skill folder and no routing
line. The arm identity keys the ledger attempt, test name, row and summary;
candidate folders are named after the fixture project and the seed commit
has a neutral author. summarize reads one campaign (run id and PREREG.md
sha256) and refuses a duplicate cell; suggestionLine reads the skill arms
only, so the preregistered particle-1 line is unchanged. renderResults writes
results.md, which passes gstack-eval-plan results --check against the new
test/fixtures/model-panel/PREREG.md (supersedes GSTA-173 for later runs).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…, both-order paired judge, spend ledger (GST-469)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…470)

GSTACK_SAFETY_ARM and GSTACK_ROUTER_ARM take a third value, none: the same
fixture, tools and user request with no skill excerpt, rule text, step file
or installed skill. Under none every behavioral check is recorded as an
observation (none-arm-observations.jsonl) and never fails the case or writes
contract-violations.jsonl; harness preconditions still assert.

The four safety prompt builders (ship-evidence, codex-boundary,
codex-consult, design-risk) and the router fixtures move into helpers; the
rule, removed, current and before prompts stay byte-identical, proven
against reference copies of main's code. pair-agent and ios-demo refuse
none with the reason (NO_NONE_ARM). GSTACK_ROUTER_ARM now throws on an
unknown value instead of silently running current.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…/6, $7.22)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…ctable by --case

The lanes moved each cell's testName into helpers, so the static detector no
longer saw a computed registration and B4 failed for all eight cases. Same
runtime test names; the paid runs are unaffected.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

garrytan and others added 8 commits October 31, 2026 17:36
…for the no-gstack arms

scripts/judge-calibration.ts is now an arm-benchmark paid touchfile, so it can no
longer be a free consumer of test/fixtures/judge-calibration/** (CI free-suite 8).
The eval-plan Step 5 fence names <prereg-path> and <results-path>; allowlist both
as PATH placeholders (CI free-suite 19).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…utput and the results usage error

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
… corpus

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@github-actions github-actions Bot changed the title evals: no-gstack arms for the model panel, arm-benchmark, safety and routing evals v1.92.2.0 evals: no-gstack arms for the model panel, arm-benchmark, safety and routing evals Oct 31, 2026
@capy-ai
capy-ai Bot merged commit 4eff47c into main Oct 31, 2026
62 checks passed
garrytan added a commit that referenced this pull request Oct 31, 2026
Resolves bin/gstack-eval-plan help and docs/skills.md by keeping both sides (#3708's results --check and this PR's per-arm report and --seat-metered text). Raises the eval-plan eager context ceiling 4887 -> 5021 (+134 tok): #3708 and this PR each fit alone; together eval-plan/SKILL.md measures 5021.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant