Repository navigation
Stop the review coverage eval failing when zsh cuts a file-read line short (GST-2703) - #4364
Merged
Merged
Conversation
… separator (GST-2703) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
|
😎 This pull request was merged. |
Merged
capy-ai Bot
added a commit
that referenced
this pull request
Nov 17, 2026
- /autoplan's approval page no longer hides a reviewer's product objection behind "keep your direction". (#4431) - Every skill that loads the shared preamble, on every host, now says to ask a question once and to check a file before handing it over. (#4383) - The Captain manuals now name the `Failed or skipped:` and `Also changed:` lines that READY and lane reports carry. (#4432) - gstack's shell locks hold again on Ubuntu 26.04, where `mkdir` reports success when another process won the race (GST-2247). (#4262) - `test/gstack-lock-race.test.ts` runs the old pattern and the helper against a PATH shim that behaves like uutils 0.8.0 `mkdir`, so it runs on any runner image: the old pattern lets two of four racers win, the helper admits exactly one in each of 60 races and never overlaps across take, hold and release loops, and parallel `gstack-config` writers keep every key. (#4262) - Windows Setup E2E now runs when `bin/gstack-lock.sh` changes. (#4433) - Claude Code's skill list now says when to use each gstack skill, so Opus loads /review and /plan-eng-review when a request fits. (#4387) - A pull request's merge head can no longer keep its `/ship` receipt without a full review. (#4365) - `/ship` now counts review rounds per pull request, and the head that merges always gets a full review. (#4360) - /ship's documentation audit no longer accepts a passing result when an earlier line from the same audit said "blocked". (#4434) - /ship's documentation audit no longer spends an attempt when the audit child writes text after its result. (#4285) - The README lists the two new guides, and the install doc counts its hooks right. (#4403) - docs/releases.md now says the stable release cut treats a failed commit status as red and a pending one as still going, as it has since #4257. (#4436) - /ship's `finish --docs-requested` now accepts a docs fix when the audit named the file with a line number. (#4437) - Document the `.gstack/approval-classes.json` schema, `gstack-approval-classes` CLI commands, and owner approval citation format in `docs/ship-policy.md#approval-classes`. (#4355) - `/eval-plan` now runs four result checks instead of leaving them to judgment. (#4384) - The /review coverage-audit eval no longer fails a run that really read the source file when zsh stops a command line at an `echo ====` separator. (#4364) - Sixteen in-repo documentation links reach their section again. (#4394) - gstack agents now open every status report with what failed, what they didn't verify and what else they changed. (#4273) - Two new short guides. (#4380) - A skill you wrote whose contract still lists `check_after_last_edit` keeps the rest of its contract. (#4439) - The ship policy guide now explains how to merge any pull request with `gstack-ship-queue land`, even in a repository with one maintainer. (#4377) - /ship's test step runs a large unit suite on Linux. (#4426) - Every skill that loads the shared preamble, on every host, now says never to print a whole environment, git config or process table. (#4382) - Pull requests from /ship now open with at most three plain lines, and every merge lands as one commit titled with the PR title. (#4438) - `gstack-pr-wait`, `gstack-ci-gate` and the release cut no longer call a commit green while a check on it failed. (#4257) - `gstack-runaway` now says what a thread's cold and empty wakes cost, not just how many there were (GST-2770). (#4385) - The optional Stop hook no longer checks for a test run after the last edit, and /verify asks for it in plain words instead. (#4428) - /ship's test step says when it ran out of time, and you can set its budget. (#4429) - The remote test command no longer puts every file on one command line. (#4429) - The stable release no longer waits on the nightly Windows tests (GST-2815). (#4400) - Ubicloud fan-outs can no longer overshoot the vCPU limit, and owners no longer delete each other's VMs. (#4430) - hono is pinned to 4.13.16, which fixes five advisories the weekly OSV scan found in 4.13.13 (gstack#4406). (#4408) - `package.json` overrides (#4408) - `test/dependency-security.test.ts`: the lockfile floor check now covers `hono >= 4.13.16`. (#4408) - The nightly Windows test run finishes again. (#4409) - A short write-up checklist in ETHOS.md. (#4381) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
/reviewcoverage-audit eval's read check now credits a source file the model really read when zsh cuts the command line short at anecho ====separator, so the case stops going red at random on main's gate census. That serves the goal of a trustworthy paid gate: a red should mean the skill misbehaved, not that the checker misread the transcript.What changed
What changed.
test/helpers/coverage-audit-evidence.ts: when the normal read check credits nothing and the line has a standaloneecho =wordafter;or&&, the checker matches the commands before that echo, under the same rules as before, followed by exactly the zsh error line, which must end the output. Nothing after the echo ran, so it needs no grammar. Under bash,echo ====prints====and the line continues, so this path never matches there.Tests.
test/coverage-audit-evidence.test.tsreplays the two captured calls byte for byte. On main the full recorded transcript givessourceRead: falsewith the recorded failure; with this change it passes. The test display after the separator is still not credited: the abandoned Bash call alone still fails "missing successful test-file read". These controls still get no credit: output after the error, a different error line, a line that ran on, a failed result, quoted or non-leading=words, a piped, redirected or backgrounded separator,||,printf, a tampered marker, a truncated file, a foreign or relative path, a substitution, and an unsupported command before the read. zsh's behavior was checked locally (zsh -c 'eval "printf src\\n; echo ====; echo after"'printssrcand the error, neverafter).Why
"missing successful source-file read" was the failure census run 43575953190 reported, even though the model had printed the whole source file. In census run 43575953190 (main 3b65035, slice 15), the model ran:
The runner's shell is zsh. zsh treats an unquoted word that starts with
=as a command lookup, soecho ====looked for a command named===, printed(eval):1: === not foundand abandoned the rest of the line. The output was the complete numbered source file (with its per-run marker) followed by that one error line. The model then read the test file with Read. The checker refused the whole Bash call, partly becausegit statusis outside its closed display grammar, so it reported "missing successful source-file read". The in-run rerun passed.Blast radius
Only the paid coverage-audit evals read this helper (review-coverage-audit, ship-coverage-audit, plan-eng-coverage-audit and their free AF tests). The new path runs only when the normal check credited nothing, and it still requires the exact printed bytes and the per-run marker, so it cannot credit a read that did not happen. The adjacent caller tests (test/coverage-audit.test.ts, test/ship-coverage-audit-af.test.ts) pass 45 of 45, and CI's free suite ran 37,307 tests green at this tree in run 43604842008.
Pre-Landing Review
Pre-Landing Review: No issues found.
Exploratory QA
Smoke, 2 probes, pass. The adjacent callers passed 45 of 45. Owned-fixture probes of the abort path: both files printed before the separator (with
;and with&&) are both credited, a trailing newline after the zsh error still credits the source, and a bash-printed separator with an unsupported tail credits nothing.Scope Drift
Scope Check: CLEAN
Documentation
Status: current. No documentation edits were needed.
Scope: commit 1d547b3 against base 3c15b72 on fix-coverage-audit-zsh-separator:
test/helpers/coverage-audit-evidence.ts,test/coverage-audit-evidence.test.tsandchangelog.d/fix-coverage-audit-zsh-separator.md. There were no staged, unstaged or new files. I searched the repo's docs and templates for coverage-audit references.Documentation health:
changelog.d/fix-coverage-audit-zsh-separator.md: current. It describes the zshecho ====fix accurately.docs/TEST_PORTFOLIO.md: current. Its test mapping is unchanged.docs/evals/detector-inventory.md: stale line citations. These were already wrong before this branch (see debt).Coverage debt: none. The change only affects internal test-harness behavior and adds no public surface.
Diagram drift: none.
Pre-existing debt:
docs/evals/detector-inventory.md:82citescoverage-audit-evidence.ts:546-547and:459-510. Those numbers did not match the helper's code at the base commit either, and this change moves the code another 24 lines down.Test plan
bun test test/coverage-audit-evidence.test.ts: 201 pass, 0 failbun run test: carried from CI free-tests run 43604842008 at tree abfaaaca599a, 37,307 testsAgent notes
wc -cin v1.93.2.0 GSTA-240 follow-up: native cookie cleanup waits for Edge to exit, plus three flaky-test repairs #3805). Same zsh=mechanism as the docsync fix in v1.92.31.0 fix(flaky): qa-callers child handoff and three ship-docsync detector gaps (2026-10-29 flaky pass, GSTA-162) #3606 (docsCompletedRead).test/helpers/free-fixtures-data.ts, a global paid-runner input that selects about 55 PR probes; inlined, the PR selects 5 (gstack-upgrade-happy-path, plan-eng-coverage-audit, review-coverage-audit, ship-coverage-audit, ship-local-workflow).bun run typecheckclean;typecheck:testnone new. No paid run from this lane.Approvals
No approvals recorded for branch fix-coverage-audit-zsh-separator.
Plan: exempt: bug fix
Plan check: exempt bug fix