Skip to content

Make /eval-plan check judges, zeros, failure counts and decisions before results count (GST-2754) - #4384

Merged
capy-ai[bot] merged 3 commits into
mainfrom
eval-plan-result-checks
Nov 16, 2026
Merged

capy-ai[bot] merged 3 commits into
mainfrom
eval-plan-result-checks

Conversation

@garrytan

@garrytan garrytan commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

/eval-plan now runs four checks before a result counts: a model judge has to agree with hand labels, a zero has to come from a counter that was shown to find a planted case, a failure count has to sit beside the answer-only count, and the plan has to say what each result changes before any money is spent.

Why: in our own agent teams, a counter that never fired reported zero, and a reported rise in failures disappeared once the grader's own rule counted it.

Nothing to do: preregistrations frozen before 2026-11-15 keep the rules they were frozen under.

What changed
  • prereg --check now requires ## Decisions (two or more if <result>: <action> lines with different actions; this replaces Bars' unchecked "what a miss means for the decision" clause) and ## Scoring (judge: none, or a judge model from another family than every arm, with labels: ≥ 40 and agreement: ≥ 0.6; with two or more arms it also needs paired: yes, interval: 95% and mdd:). New logic lives in lib/eval-plan-checks.ts.
  • New judge --check <prereg> --labels <jsonl>: computes Cohen's kappa of the judge against {"id","human","judge"} rows and refuses with EVAL_JUDGE_UNCHECKED.
  • results --check now takes --labels and refuses with one code per kind: EVAL_JUDGE_UNCHECKED, EVAL_REPORT_UNPLANTED_ZERO (a zero in the arm table whose column isn't a ## Diagnostics counter with expected ≥ 1) and EVAL_REPORT_NO_ANSWER_ONLY (a failures column with no answer-only column in the same table). The existing ## Diagnostics planted-fixture run now backs the zero check.
  • A frozen: YYYY-MM-DD line before 2026-11-15 exempts a preregistration from all four checks (result_checks=exempt). The two frozen fixture preregistrations (arm-benchmark, model-panel) get frozen: 2026-10-31, and their bars, arms, decoys and stop rules are unchanged.
  • Docs: docs/TESTING_INTERNALS.md#result-checks, three new troubleshooting anchors, an updated PREREG_INCOMPLETE entry, and eval-plan/SKILL.md.tmpl Steps 1, 4 and 5, regenerated.
  • Test fixtures gain the two new sections (present=6/6 becomes 8/8). <labels-path> is added to the placeholder allowlist, and lib/eval-plan-checks.ts to the exercised-map REVIEWED_SUPERSET.
Documentation

Status: updated. One factual doc edit; no blockers.

Scope: the eval-plan-result-checks candidate (0326b144..c2a5fe9): 20 committed files and no staged, unstaged or new files. It adds ## Decisions and ## Scoring preregistration checks, gstack-eval-plan judge --check, and three new result refusals: EVAL_JUDGE_UNCHECKED, EVAL_REPORT_UNPLANTED_ZERO and EVAL_REPORT_NO_ANSWER_ONLY.

Documentation health:

  • docs/skills.md: updated. The /eval-plan section now covers Decisions and Scoring, judge --check and the new results --check refusals.
  • docs/TESTING_INTERNALS.md: current. The new #result-checks section matches the code.
  • docs/troubleshooting.md: current. The three new anchors and the expanded PREREG_INCOMPLETE entry match the code.
  • eval-plan/SKILL.md.tmpl: current.
  • changelog.d/eval-plan-result-checks.md: current.
  • README.md: current. Its summary row lists only some sections, which is still accurate.

Coverage debt: judge --check and the result checks have reference docs (troubleshooting, TESTING_INTERNALS, skills.md) and how-to steps in the skill. There is no tutorial: no worked example of a labels JSONL file or a results table with an answer-only column. That is a common gap; /document-generate could fill it.

Diagram drift: none. No diagrams cover eval-plan.

Review notes

Below the blocking floor (advisory, informational, test-gap or wording items); recorded here and not fixed in this PR.

  • INFORMATIONAL (no path) 1. An untested diagnostic declaration authorizes verified zeros. At lib/eval-plan-report.ts:68–71, planted comes solely from declared expected counts, without evidence that the fixture exists or the summarizer passed. Reproduced checkResults(...).ok === true for a zero-valued timeouts co
  • INFORMATIONAL (no path) 2. Omitting Scoring and Decisions bypasses the new result checks. At lib/eval-plan-report.ts:60–63, only section-specific errors are propagated; absent or empty sections return empty error arrays. The general prereg.ok check applies only when stack is not none. Reproduced a non-grandfath
  • INFORMATIONAL (no path) 3. An empty answer-only column satisfies the failure-comparison gate. At lib/eval-plan-report.ts:75, only the header is checked. Reproduced acceptance of a table containing failures | answer-only with values 3 | —; omitted answer-only cells also escape validation. Reports can therefore pub
  • INFORMATIONAL (no path) 4. Floating-point error rejects judges exactly at the passing bar. At lib/eval-plan-checks.ts:117, the raw floating-point kappa is compared directly against the threshold. A 40-row confusion matrix with counts (pass/pass, pass/fail, fail/pass, fail/fail) = (9, 1, 6, 24) has exact kappa 0.6
  • INFORMATIONAL (no path) 1. FIXABLE: the same-family judge rule can be bypassed with short model names. In lib/eval-plan-checks.ts:35, modelFamily maps only claude, gpt, o, codex, gemini, gemma, grok, llama and muse to a lab. Anything else becomes its own first word. So judge: opus-5-5 against arm
  • INFORMATIONAL (no path) 2. INVESTIGATE: the zero check refuses zeros that never came from a counter, including null results. In lib/eval-plan-report.ts:71, every column after the first, in every table of the results file, is refused when any cell is 0, 0% or 0/N. That includes a cost USD column (my probe return
  • INFORMATIONAL (no path) 3. INVESTIGATE: the frozen-date exemption is self-declared and only accepts one exact format. At lib/eval-plan-checks.ts:26-29, adding a frozen: 2026-01-01 line anywhere in the file switches off all four checks; my probe showed a prereg with no Decisions section passing. The regex also accepts
  • INFORMATIONAL (no path) 4. INVESTIGATE: the planted-zero check only reads declarations. At lib/eval-plan-report.ts:68, a column counts as "planted" when Diagnostics declares a counter with expected >= 1. results --check never confirms the summarizer ran or that its PREREG_DIAGNOSTIC line said ok. The docs say the
  • INFORMATIONAL (no path) 5. FIXABLE: the failures-column regex is too broad. At lib/eval-plan-report.ts:74, /\bfail/i matches a Pass/Fail verdict header (my probe returned EVAL_REPORT_NO_ANSWER_ONLY) and words like failover. Severity: Low. Blocks: no.
  • INFORMATIONAL (no path) 6. FIXABLE: one prereg failure gets reported under two codes. At lib/eval-plan-report.ts:54, a prereg that fails only on Decisions or Scoring also emits EVAL_REPORT_NO_GSTACK_ROW: the preregistration does not pass prereg --check next to PREREG_INCOMPLETE. That sends the user to the wrong tro
  • INFORMATIONAL (no path) 7. FIXABLE: two judge --check edge cases. At bin/gstack-eval-plan:171, the command scores labels even when scoring.errors is non-empty, so a same-family judge can print JUDGE_AGREEMENT: ok, a line the approval ask quotes. And on an exempt prereg that does name a judge, it exits 2 with "nee
  • INFORMATIONAL (no path) 8. INVESTIGATE: the labels file can't be verified. Nothing ties the judge column to real outputs from the preregistered judge, or shows the ids aren't held-out rows. The kappa is only as honest as whoever wrote the file. This is by design, but it should be stated. Severity: Low. Blocks: no.
Agent notes

Open notes (below the blocking floor; not fixed in this PR):

  1. The stack: none hole and its two-line fix: a preregistration that leaves out ## Decisions and ## Scoring passes results --check (lib/eval-plan-report.ts:60). Fix: report missing or empty Decisions and Scoring as PREREG_INCOMPLETE whatever the stack, plus one test.
  2. A declared Diagnostics counter exempts a zero even when its fixture and summarizer don't exist; results --check never confirms the diagnostic ran (lib/eval-plan-report.ts:68).
  3. An answer-only column filled with dashes or empty cells passes (lib/eval-plan-report.ts:75).
  4. Kappa exactly at the bar is falsely refused, a known false refusal (lib/eval-plan-checks.ts:117), e.g. 45 labels at exact kappa 0.6 computes 0.5999999999999999. Fix: compare with a small epsilon or on rounded values.
  5. The opus-5-5 / fable-5-2 alias gap: modelFamily maps no product aliases, so a judge written as opus-5-5 or fable-5-2 passes as cross-family for a Claude arm (lib/eval-plan-checks.ts:35).

No paid eval ran for this PR; the paid release gate covers /eval-plan's live probes.

Review: round 2 of 2 (R51). GStack Minder ruled ship with open notes (approvals a-e8851e97ee42 and a-81cc81c81550). The follow-up PR for the five notes is queued post-freeze.

Follow-up measure (R56). Re-score one finished gstack eval's judge against 40 hand labels with judge --check, and record its kappa against the 0.6 bar. Not run in this PR.

Checks run locally before the push: typecheck, the typecheck:test ratchet (no new diagnostics), parity-suite, the context-budget ratchet (eval-plan is under its 5700-token ceiling after trimming), env-host fence lint, generated-bash placeholders, exercised-map, every eval-plan test and the new test/eval-plan-result-checks.test.ts (14 tests). Two local guard failures this change caused (fence placeholder lint, exercised-map superset) were fixed. The rest were 5-second timeouts in ship-receipt, review-notes and ship-skip-actor tests that also time out on clean origin/main on the 4-core machine. CI at the final head is the full run, and it passed.

CI at dd4f0e8: 41 passed, 0 failed, 0 pending, 13 skipped.

Approvals

Approved: yes (record r-d87ecb5605fb). The full approval record stays private.
Plan: exempt: audited-list
Plan check: exempt audited-list

REVIEW_LOOP: passes=1 blocking_found=4 deferred=0 noted=0 fixed=0 scope=frozen unrecorded=4 unbound=1 ledgers_read=2 dispatches=1

{
  "schema_version": 1,
  "pr": 4384,
  "head": "dd4f0e84b8473c0a2dbae79a2f1ac6a5c32f3823",
  "version": "1.99.5.0",
  "base": "main",
  "gate": {
    "free": {
      "pass": 1670,
      "fail": 0,
      "skip": 0
    }
  },
  "ci_carried": [
    {
      "lane": "free",
      "run": 43679574306,
      "attempt": 1,
      "tree": "4667c9686c51e86f6a7861da8c66c466cfa5e8b7"
    }
  ],
  "session_kind": "unattended",
  "head_tree": "4667c9686c51e86f6a7861da8c66c466cfa5e8b7",
  "product_patch_id": "11538364a9070fc103cfc5ec78c8e861dc5bc0a2",
  "gstack_version": "1.99.5.0",
  "review_depth": {
    "depth": "light",
    "class": "code",
    "runtime_lines": 346,
    "rules": "66e5174161eecc2c",
    "base": "0326b144af36676d599dcf63a9fad9a870d22ed6",
    "registry": "0326b144af36676d599dcf63a9fad9a870d22ed6",
    "gate_scan": {
      "tagged": 0,
      "head": "dd4f0e84b8473c0a2dbae79a2f1ac6a5c32f3823"
    },
    "reasons": [
      "runtime=346"
    ],
    "risk": [],
    "mode": "panel"
  },
  "pregate": "requires-remote cleared @4667c9686c51 (8 checks, current for dd4f0e84b847)",
  "requires_remote": [
    "free-tests.yml/cso-macos-launcher: https://github.com/garrytan/gstack/actions/runs/43679574306/attempts/1",
    "free-tests.yml/macos-named-regressions: https://github.com/garrytan/gstack/actions/runs/43679574306/attempts/1",
    "windows-smoke.yml/windows-smoke: https://github.com/garrytan/gstack/actions/runs/43679574285/attempts/1",
    "pregate.json/remote_gate: remote-gate tree=4667c9686c51 tests=37327 skipped=106 runs=43679574306/1 profiles=ci-reuse/free-tests heads=dd4f0e84b847 verified=2026-11-15T05:17:17.454Z"
  ],
  "issues": {
    "fixes": [],
    "refs": [],
    "flakes": []
  },
  "docs": {
    "status": "updated",
    "scope": "docs/, eval-plan/SKILL.md.tmpl, README.md for the eval-plan result checks"
  },
  "review_loop": {
    "passes": 1,
    "blocking_found": 4,
    "deferred": 0,
    "noted": 0,
    "fixed": 0,
    "scope": "frozen",
    "rounds": 1,
    "unrecorded": 4,
    "ledgers_read": 2,
    "unbound": 1,
    "dispatches": {
      "specialist": 1
    }
  },
  "gate_dispositions": []
}

Open in Capy

…ore results count (GST-2754)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

garrytan and others added 2 commits November 15, 2026 04:47
…stack: none, count only valid judge labels (GST-2754)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@capy-ai
capy-ai Bot merged commit fdf3f1e into main Nov 16, 2026
58 checks passed
@garrytan garrytan mentioned this pull request Nov 17, 2026
capy-ai Bot added a commit that referenced this pull request Nov 17, 2026
- /autoplan's approval page no longer hides a reviewer's product objection behind "keep your direction". (#4431)
- Every skill that loads the shared preamble, on every host, now says to ask a question once and to check a file before handing it over. (#4383)
- The Captain manuals now name the `Failed or skipped:` and `Also changed:` lines that READY and lane reports carry. (#4432)
- gstack's shell locks hold again on Ubuntu 26.04, where `mkdir` reports success when another process won the race (GST-2247). (#4262)
- `test/gstack-lock-race.test.ts` runs the old pattern and the helper against a PATH shim that behaves like uutils 0.8.0 `mkdir`, so it runs on any runner image: the old pattern lets two of four racers win, the helper admits exactly one in each of 60 races and never overlaps across take, hold and release loops, and parallel `gstack-config` writers keep every key. (#4262)
- Windows Setup E2E now runs when `bin/gstack-lock.sh` changes. (#4433)
- Claude Code's skill list now says when to use each gstack skill, so Opus loads /review and /plan-eng-review when a request fits. (#4387)
- A pull request's merge head can no longer keep its `/ship` receipt without a full review. (#4365)
- `/ship` now counts review rounds per pull request, and the head that merges always gets a full review. (#4360)
- /ship's documentation audit no longer accepts a passing result when an earlier line from the same audit said "blocked". (#4434)
- /ship's documentation audit no longer spends an attempt when the audit child writes text after its result. (#4285)
- The README lists the two new guides, and the install doc counts its hooks right. (#4403)
- docs/releases.md now says the stable release cut treats a failed commit status as red and a pending one as still going, as it has since #4257. (#4436)
- /ship's `finish --docs-requested` now accepts a docs fix when the audit named the file with a line number. (#4437)
- Document the `.gstack/approval-classes.json` schema, `gstack-approval-classes` CLI commands, and owner approval citation format in `docs/ship-policy.md#approval-classes`. (#4355)
- `/eval-plan` now runs four result checks instead of leaving them to judgment. (#4384)
- The /review coverage-audit eval no longer fails a run that really read the source file when zsh stops a command line at an `echo ====` separator. (#4364)
- Sixteen in-repo documentation links reach their section again. (#4394)
- gstack agents now open every status report with what failed, what they didn't verify and what else they changed. (#4273)
- Two new short guides. (#4380)
- A skill you wrote whose contract still lists `check_after_last_edit` keeps the rest of its contract. (#4439)
- The ship policy guide now explains how to merge any pull request with `gstack-ship-queue land`, even in a repository with one maintainer. (#4377)
- /ship's test step runs a large unit suite on Linux. (#4426)
- Every skill that loads the shared preamble, on every host, now says never to print a whole environment, git config or process table. (#4382)
- Pull requests from /ship now open with at most three plain lines, and every merge lands as one commit titled with the PR title. (#4438)
- `gstack-pr-wait`, `gstack-ci-gate` and the release cut no longer call a commit green while a check on it failed. (#4257)
- `gstack-runaway` now says what a thread's cold and empty wakes cost, not just how many there were (GST-2770). (#4385)
- The optional Stop hook no longer checks for a test run after the last edit, and /verify asks for it in plain words instead. (#4428)
- /ship's test step says when it ran out of time, and you can set its budget. (#4429)
- The remote test command no longer puts every file on one command line. (#4429)
- The stable release no longer waits on the nightly Windows tests (GST-2815). (#4400)
- Ubicloud fan-outs can no longer overshoot the vCPU limit, and owners no longer delete each other's VMs. (#4430)
- hono is pinned to 4.13.16, which fixes five advisories the weekly OSV scan found in 4.13.13 (gstack#4406). (#4408)
- `package.json` overrides (#4408)
- `test/dependency-security.test.ts`: the lockfile floor check now covers `hono >= 4.13.16`. (#4408)
- The nightly Windows test run finishes again. (#4409)
- A short write-up checklist in ETHOS.md. (#4381)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant