Repository navigation
Make /eval-plan check judges, zeros, failure counts and decisions before results count (GST-2754) - #4384
Merged
Conversation
…ore results count (GST-2754) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
|
😎 This pull request was merged. |
…stack: none, count only valid judge labels (GST-2754) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
4 of 5 tasks
Merged
capy-ai Bot
added a commit
that referenced
this pull request
Nov 17, 2026
- /autoplan's approval page no longer hides a reviewer's product objection behind "keep your direction". (#4431) - Every skill that loads the shared preamble, on every host, now says to ask a question once and to check a file before handing it over. (#4383) - The Captain manuals now name the `Failed or skipped:` and `Also changed:` lines that READY and lane reports carry. (#4432) - gstack's shell locks hold again on Ubuntu 26.04, where `mkdir` reports success when another process won the race (GST-2247). (#4262) - `test/gstack-lock-race.test.ts` runs the old pattern and the helper against a PATH shim that behaves like uutils 0.8.0 `mkdir`, so it runs on any runner image: the old pattern lets two of four racers win, the helper admits exactly one in each of 60 races and never overlaps across take, hold and release loops, and parallel `gstack-config` writers keep every key. (#4262) - Windows Setup E2E now runs when `bin/gstack-lock.sh` changes. (#4433) - Claude Code's skill list now says when to use each gstack skill, so Opus loads /review and /plan-eng-review when a request fits. (#4387) - A pull request's merge head can no longer keep its `/ship` receipt without a full review. (#4365) - `/ship` now counts review rounds per pull request, and the head that merges always gets a full review. (#4360) - /ship's documentation audit no longer accepts a passing result when an earlier line from the same audit said "blocked". (#4434) - /ship's documentation audit no longer spends an attempt when the audit child writes text after its result. (#4285) - The README lists the two new guides, and the install doc counts its hooks right. (#4403) - docs/releases.md now says the stable release cut treats a failed commit status as red and a pending one as still going, as it has since #4257. (#4436) - /ship's `finish --docs-requested` now accepts a docs fix when the audit named the file with a line number. (#4437) - Document the `.gstack/approval-classes.json` schema, `gstack-approval-classes` CLI commands, and owner approval citation format in `docs/ship-policy.md#approval-classes`. (#4355) - `/eval-plan` now runs four result checks instead of leaving them to judgment. (#4384) - The /review coverage-audit eval no longer fails a run that really read the source file when zsh stops a command line at an `echo ====` separator. (#4364) - Sixteen in-repo documentation links reach their section again. (#4394) - gstack agents now open every status report with what failed, what they didn't verify and what else they changed. (#4273) - Two new short guides. (#4380) - A skill you wrote whose contract still lists `check_after_last_edit` keeps the rest of its contract. (#4439) - The ship policy guide now explains how to merge any pull request with `gstack-ship-queue land`, even in a repository with one maintainer. (#4377) - /ship's test step runs a large unit suite on Linux. (#4426) - Every skill that loads the shared preamble, on every host, now says never to print a whole environment, git config or process table. (#4382) - Pull requests from /ship now open with at most three plain lines, and every merge lands as one commit titled with the PR title. (#4438) - `gstack-pr-wait`, `gstack-ci-gate` and the release cut no longer call a commit green while a check on it failed. (#4257) - `gstack-runaway` now says what a thread's cold and empty wakes cost, not just how many there were (GST-2770). (#4385) - The optional Stop hook no longer checks for a test run after the last edit, and /verify asks for it in plain words instead. (#4428) - /ship's test step says when it ran out of time, and you can set its budget. (#4429) - The remote test command no longer puts every file on one command line. (#4429) - The stable release no longer waits on the nightly Windows tests (GST-2815). (#4400) - Ubicloud fan-outs can no longer overshoot the vCPU limit, and owners no longer delete each other's VMs. (#4430) - hono is pinned to 4.13.16, which fixes five advisories the weekly OSV scan found in 4.13.13 (gstack#4406). (#4408) - `package.json` overrides (#4408) - `test/dependency-security.test.ts`: the lockfile floor check now covers `hono >= 4.13.16`. (#4408) - The nightly Windows test run finishes again. (#4409) - A short write-up checklist in ETHOS.md. (#4381) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
/eval-plannow runs four checks before a result counts: a model judge has to agree with hand labels, a zero has to come from a counter that was shown to find a planted case, a failure count has to sit beside the answer-only count, and the plan has to say what each result changes before any money is spent.Why: in our own agent teams, a counter that never fired reported zero, and a reported rise in failures disappeared once the grader's own rule counted it.
Nothing to do: preregistrations frozen before 2026-11-15 keep the rules they were frozen under.
What changed
prereg --checknow requires## Decisions(two or moreif <result>: <action>lines with different actions; this replaces Bars' unchecked "what a miss means for the decision" clause) and## Scoring(judge: none, or a judge model from another family than every arm, withlabels:≥ 40 andagreement:≥ 0.6; with two or more arms it also needspaired: yes,interval: 95%andmdd:). New logic lives inlib/eval-plan-checks.ts.judge --check <prereg> --labels <jsonl>: computes Cohen's kappa of the judge against{"id","human","judge"}rows and refuses withEVAL_JUDGE_UNCHECKED.results --checknow takes--labelsand refuses with one code per kind:EVAL_JUDGE_UNCHECKED,EVAL_REPORT_UNPLANTED_ZERO(a zero in the arm table whose column isn't a## Diagnosticscounter with expected ≥ 1) andEVAL_REPORT_NO_ANSWER_ONLY(a failures column with noanswer-onlycolumn in the same table). The existing## Diagnosticsplanted-fixture run now backs the zero check.frozen: YYYY-MM-DDline before 2026-11-15 exempts a preregistration from all four checks (result_checks=exempt). The two frozen fixture preregistrations (arm-benchmark, model-panel) getfrozen: 2026-10-31, and their bars, arms, decoys and stop rules are unchanged.docs/TESTING_INTERNALS.md#result-checks, three new troubleshooting anchors, an updatedPREREG_INCOMPLETEentry, and eval-plan/SKILL.md.tmpl Steps 1, 4 and 5, regenerated.present=6/6becomes8/8).<labels-path>is added to the placeholder allowlist, andlib/eval-plan-checks.tsto the exercised-map REVIEWED_SUPERSET.Documentation
Status: updated. One factual doc edit; no blockers.
Scope: the eval-plan-result-checks candidate (
0326b144..c2a5fe9): 20 committed files and no staged, unstaged or new files. It adds## Decisionsand## Scoringpreregistration checks,gstack-eval-plan judge --check, and three new result refusals:EVAL_JUDGE_UNCHECKED,EVAL_REPORT_UNPLANTED_ZEROandEVAL_REPORT_NO_ANSWER_ONLY.Documentation health:
docs/skills.md: updated. The/eval-plansection now covers Decisions and Scoring,judge --checkand the newresults --checkrefusals.docs/TESTING_INTERNALS.md: current. The new#result-checkssection matches the code.docs/troubleshooting.md: current. The three new anchors and the expandedPREREG_INCOMPLETEentry match the code.eval-plan/SKILL.md.tmpl: current.changelog.d/eval-plan-result-checks.md: current.README.md: current. Its summary row lists only some sections, which is still accurate.Coverage debt:
judge --checkand the result checks have reference docs (troubleshooting, TESTING_INTERNALS, skills.md) and how-to steps in the skill. There is no tutorial: no worked example of a labels JSONL file or a results table with ananswer-onlycolumn. That is a common gap;/document-generatecould fill it.Diagram drift: none. No diagrams cover eval-plan.
Review notes
Below the blocking floor (advisory, informational, test-gap or wording items); recorded here and not fixed in this PR.
(no path)1. An untested diagnostic declaration authorizes verified zeros. Atlib/eval-plan-report.ts:68–71,plantedcomes solely from declared expected counts, without evidence that the fixture exists or the summarizer passed. ReproducedcheckResults(...).ok === truefor a zero-valuedtimeoutsco(no path)2. Omitting Scoring and Decisions bypasses the new result checks. Atlib/eval-plan-report.ts:60–63, only section-specific errors are propagated; absent or empty sections return empty error arrays. The generalprereg.okcheck applies only whenstackis notnone. Reproduced a non-grandfath(no path)3. An empty answer-only column satisfies the failure-comparison gate. Atlib/eval-plan-report.ts:75, only the header is checked. Reproduced acceptance of a table containingfailures | answer-onlywith values3 | —; omitted answer-only cells also escape validation. Reports can therefore pub(no path)4. Floating-point error rejects judges exactly at the passing bar. Atlib/eval-plan-checks.ts:117, the raw floating-point kappa is compared directly against the threshold. A 40-row confusion matrix with counts(pass/pass, pass/fail, fail/pass, fail/fail) = (9, 1, 6, 24)has exact kappa0.6(no path)1. FIXABLE: the same-family judge rule can be bypassed with short model names. In lib/eval-plan-checks.ts:35,modelFamilymaps onlyclaude,gpt,o,codex,gemini,gemma,grok,llamaandmuseto a lab. Anything else becomes its own first word. Sojudge: opus-5-5against arm(no path)2. INVESTIGATE: the zero check refuses zeros that never came from a counter, including null results. In lib/eval-plan-report.ts:71, every column after the first, in every table of the results file, is refused when any cell is0,0%or0/N. That includes acost USDcolumn (my probe return(no path)3. INVESTIGATE: the frozen-date exemption is self-declared and only accepts one exact format. At lib/eval-plan-checks.ts:26-29, adding afrozen: 2026-01-01line anywhere in the file switches off all four checks; my probe showed a prereg with no Decisions section passing. The regex also accepts(no path)4. INVESTIGATE: the planted-zero check only reads declarations. At lib/eval-plan-report.ts:68, a column counts as "planted" when Diagnostics declares a counter withexpected >= 1.results --checknever confirms the summarizer ran or that itsPREREG_DIAGNOSTICline said ok. The docs say the(no path)5. FIXABLE: the failures-column regex is too broad. At lib/eval-plan-report.ts:74,/\bfail/imatches aPass/Failverdict header (my probe returnedEVAL_REPORT_NO_ANSWER_ONLY) and words likefailover. Severity: Low. Blocks: no.(no path)6. FIXABLE: one prereg failure gets reported under two codes. At lib/eval-plan-report.ts:54, a prereg that fails only on Decisions or Scoring also emitsEVAL_REPORT_NO_GSTACK_ROW: the preregistration does not pass prereg --checknext toPREREG_INCOMPLETE. That sends the user to the wrong tro(no path)7. FIXABLE: twojudge --checkedge cases. At bin/gstack-eval-plan:171, the command scores labels even whenscoring.errorsis non-empty, so a same-family judge can printJUDGE_AGREEMENT: ok, a line the approval ask quotes. And on an exempt prereg that does name a judge, it exits 2 with "nee(no path)8. INVESTIGATE: the labels file can't be verified. Nothing ties thejudgecolumn to real outputs from the preregistered judge, or shows the ids aren't held-out rows. The kappa is only as honest as whoever wrote the file. This is by design, but it should be stated. Severity: Low. Blocks: no.Agent notes
Open notes (below the blocking floor; not fixed in this PR):
## Decisionsand## Scoringpassesresults --check(lib/eval-plan-report.ts:60). Fix: report missing or empty Decisions and Scoring asPREREG_INCOMPLETEwhatever the stack, plus one test.results --checknever confirms the diagnostic ran (lib/eval-plan-report.ts:68).modelFamilymaps no product aliases, so a judge written as opus-5-5 or fable-5-2 passes as cross-family for a Claude arm (lib/eval-plan-checks.ts:35).No paid eval ran for this PR; the paid release gate covers /eval-plan's live probes.
Review: round 2 of 2 (R51). GStack Minder ruled ship with open notes (approvals a-e8851e97ee42 and a-81cc81c81550). The follow-up PR for the five notes is queued post-freeze.
Follow-up measure (R56). Re-score one finished gstack eval's judge against 40 hand labels with
judge --check, and record its kappa against the 0.6 bar. Not run in this PR.Checks run locally before the push: typecheck, the typecheck:test ratchet (no new diagnostics), parity-suite, the context-budget ratchet (eval-plan is under its 5700-token ceiling after trimming), env-host fence lint, generated-bash placeholders, exercised-map, every eval-plan test and the new test/eval-plan-result-checks.test.ts (14 tests). Two local guard failures this change caused (fence placeholder lint, exercised-map superset) were fixed. The rest were 5-second timeouts in ship-receipt, review-notes and ship-skip-actor tests that also time out on clean origin/main on the 4-core machine. CI at the final head is the full run, and it passed.
CI at dd4f0e8: 41 passed, 0 failed, 0 pending, 13 skipped.
Approvals
Approved: yes (record r-d87ecb5605fb). The full approval record stays private.
Plan: exempt: audited-list
Plan check: exempt audited-list
REVIEW_LOOP: passes=1 blocking_found=4 deferred=0 noted=0 fixed=0 scope=frozen unrecorded=4 unbound=1 ledgers_read=2 dispatches=1