Skip to content

Stop the review coverage eval failing when zsh cuts a file-read line short (GST-2703) - #4364

Merged
capy-ai[bot] merged 1 commit into
mainfrom
fix-coverage-audit-zsh-separator
Nov 17, 2026
Merged

capy-ai[bot] merged 1 commit into
mainfrom
fix-coverage-audit-zsh-separator

Conversation

@garrytan

@garrytan garrytan commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

The /review coverage-audit eval's read check now credits a source file the model really read when zsh cuts the command line short at an echo ==== separator, so the case stops going red at random on main's gate census. That serves the goal of a trustworthy paid gate: a red should mean the skill misbehaved, not that the checker misread the transcript.

What changed

What changed. test/helpers/coverage-audit-evidence.ts: when the normal read check credits nothing and the line has a standalone echo =word after ; or &&, the checker matches the commands before that echo, under the same rules as before, followed by exactly the zsh error line, which must end the output. Nothing after the echo ran, so it needs no grammar. Under bash, echo ==== prints ==== and the line continues, so this path never matches there.

Tests. test/coverage-audit-evidence.test.ts replays the two captured calls byte for byte. On main the full recorded transcript gives sourceRead: false with the recorded failure; with this change it passes. The test display after the separator is still not credited: the abandoned Bash call alone still fails "missing successful test-file read". These controls still get no credit: output after the error, a different error line, a line that ran on, a failed result, quoted or non-leading = words, a piped, redirected or backgrounded separator, ||, printf, a tampered marker, a truncated file, a foreign or relative path, a substitution, and an unsupported command before the read. zsh's behavior was checked locally (zsh -c 'eval "printf src\\n; echo ====; echo after"' prints src and the error, never after).

Why

"missing successful source-file read" was the failure census run 43575953190 reported, even though the model had printed the whole source file. In census run 43575953190 (main 3b65035, slice 15), the model ran:

cd <fixture>; cat -n src/billing.ts; echo ====; cat -n test/billing.test.ts; git status --short; git diff HEAD --stat

The runner's shell is zsh. zsh treats an unquoted word that starts with = as a command lookup, so echo ==== looked for a command named ===, printed (eval):1: === not found and abandoned the rest of the line. The output was the complete numbered source file (with its per-run marker) followed by that one error line. The model then read the test file with Read. The checker refused the whole Bash call, partly because git status is outside its closed display grammar, so it reported "missing successful source-file read". The in-run rerun passed.

Blast radius

Only the paid coverage-audit evals read this helper (review-coverage-audit, ship-coverage-audit, plan-eng-coverage-audit and their free AF tests). The new path runs only when the normal check credited nothing, and it still requires the exact printed bytes and the per-run marker, so it cannot credit a read that did not happen. The adjacent caller tests (test/coverage-audit.test.ts, test/ship-coverage-audit-af.test.ts) pass 45 of 45, and CI's free suite ran 37,307 tests green at this tree in run 43604842008.

Pre-Landing Review

Pre-Landing Review: No issues found.

Exploratory QA

Smoke, 2 probes, pass. The adjacent callers passed 45 of 45. Owned-fixture probes of the abort path: both files printed before the separator (with ; and with &&) are both credited, a trailing newline after the zsh error still credits the source, and a bash-printed separator with an unsupported tail credits nothing.

Scope Drift

Scope Check: CLEAN

Documentation

Status: current. No documentation edits were needed.

Scope: commit 1d547b3 against base 3c15b72 on fix-coverage-audit-zsh-separator: test/helpers/coverage-audit-evidence.ts, test/coverage-audit-evidence.test.ts and changelog.d/fix-coverage-audit-zsh-separator.md. There were no staged, unstaged or new files. I searched the repo's docs and templates for coverage-audit references.

Documentation health:

  • changelog.d/fix-coverage-audit-zsh-separator.md: current. It describes the zsh echo ==== fix accurately.
  • docs/TEST_PORTFOLIO.md: current. Its test mapping is unchanged.
  • docs/evals/detector-inventory.md: stale line citations. These were already wrong before this branch (see debt).

Coverage debt: none. The change only affects internal test-harness behavior and adds no public surface.

Diagram drift: none.

Pre-existing debt: docs/evals/detector-inventory.md:82 cites coverage-audit-evidence.ts:546-547 and :459-510. Those numbers did not match the helper's code at the base commit either, and this change moves the code another 24 lines down.

Test plan

  • bun test test/coverage-audit-evidence.test.ts: 201 pass, 0 fail
  • bun run test: carried from CI free-tests run 43604842008 at tree abfaaaca599a, 37,307 tests
Agent notes

Approvals

No approvals recorded for branch fix-coverage-audit-zsh-separator.
Plan: exempt: bug fix
Plan check: exempt bug fix

{
  "schema_version": 1,
  "pr": 4364,
  "head": "1d547b3e21206e4f682ef647efb46baffd30acd4",
  "version": "1.99.3.0",
  "base": "main",
  "gate": {
    "targeted": {
      "pass": 201,
      "fail": 0,
      "skip": 0
    }
  },
  "ci_carried": [
    {
      "lane": "free",
      "run": 43604842008,
      "attempt": 1,
      "tree": "abfaaaca599a78ec29ebaff02d579964d741fa92"
    }
  ],
  "session_kind": "unattended",
  "head_tree": "abfaaaca599a78ec29ebaff02d579964d741fa92",
  "product_patch_id": "none",
  "gstack_version": "1.99.4.0",
  "review_depth": {
    "depth": "light",
    "class": "tests",
    "runtime_lines": 0,
    "rules": "66e5174161eecc2c",
    "base": "81fb9008351d29fccc3e146d860a9cda8c104a2d",
    "registry": "81fb9008351d29fccc3e146d860a9cda8c104a2d",
    "gate_scan": {
      "tagged": 0,
      "head": "1d547b3e21206e4f682ef647efb46baffd30acd4"
    },
    "reasons": [
      "runtime=0"
    ],
    "risk": [],
    "mode": "panel"
  },
  "pregate": "requires-remote cleared @abfaaaca599a (8 checks, current for 1d547b3e2120)",
  "requires_remote": [
    "free-tests.yml/macos-named-regressions: https://github.com/garrytan/gstack/actions/runs/43604842008/attempts/1",
    "pregate.json/remote_gate: remote-gate tree=abfaaaca599a tests=37307 skipped=68 runs=43604842008/1 profiles=ci-reuse/free-tests heads=1d547b3e2120 verified=2026-11-14T14:34:59.136Z"
  ],
  "issues": {
    "fixes": [],
    "refs": [],
    "flakes": []
  },
  "docs": {
    "status": "current",
    "scope": "docs/, README.md, CONTRIBUTING.md, CLAUDE.md, AGENTS.md against base 3c15b72 (3 changed files: test helper, its test, changelog fragment)"
  },
  "review_loop": {
    "passes": 1,
    "blocking_found": 0,
    "deferred": 0,
    "noted": 0,
    "fixed": 0,
    "scope": "frozen",
    "rounds": 1,
    "ledgers_read": 1,
    "dispatches": {
      "specialist": 1
    }
  },
  "gate_dispositions": []
}

… separator (GST-2703)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

@capy-ai
capy-ai Bot merged commit 18652cc into main Nov 17, 2026
77 checks passed
@garrytan garrytan mentioned this pull request Nov 17, 2026
capy-ai Bot added a commit that referenced this pull request Nov 17, 2026
- /autoplan's approval page no longer hides a reviewer's product objection behind "keep your direction". (#4431)
- Every skill that loads the shared preamble, on every host, now says to ask a question once and to check a file before handing it over. (#4383)
- The Captain manuals now name the `Failed or skipped:` and `Also changed:` lines that READY and lane reports carry. (#4432)
- gstack's shell locks hold again on Ubuntu 26.04, where `mkdir` reports success when another process won the race (GST-2247). (#4262)
- `test/gstack-lock-race.test.ts` runs the old pattern and the helper against a PATH shim that behaves like uutils 0.8.0 `mkdir`, so it runs on any runner image: the old pattern lets two of four racers win, the helper admits exactly one in each of 60 races and never overlaps across take, hold and release loops, and parallel `gstack-config` writers keep every key. (#4262)
- Windows Setup E2E now runs when `bin/gstack-lock.sh` changes. (#4433)
- Claude Code's skill list now says when to use each gstack skill, so Opus loads /review and /plan-eng-review when a request fits. (#4387)
- A pull request's merge head can no longer keep its `/ship` receipt without a full review. (#4365)
- `/ship` now counts review rounds per pull request, and the head that merges always gets a full review. (#4360)
- /ship's documentation audit no longer accepts a passing result when an earlier line from the same audit said "blocked". (#4434)
- /ship's documentation audit no longer spends an attempt when the audit child writes text after its result. (#4285)
- The README lists the two new guides, and the install doc counts its hooks right. (#4403)
- docs/releases.md now says the stable release cut treats a failed commit status as red and a pending one as still going, as it has since #4257. (#4436)
- /ship's `finish --docs-requested` now accepts a docs fix when the audit named the file with a line number. (#4437)
- Document the `.gstack/approval-classes.json` schema, `gstack-approval-classes` CLI commands, and owner approval citation format in `docs/ship-policy.md#approval-classes`. (#4355)
- `/eval-plan` now runs four result checks instead of leaving them to judgment. (#4384)
- The /review coverage-audit eval no longer fails a run that really read the source file when zsh stops a command line at an `echo ====` separator. (#4364)
- Sixteen in-repo documentation links reach their section again. (#4394)
- gstack agents now open every status report with what failed, what they didn't verify and what else they changed. (#4273)
- Two new short guides. (#4380)
- A skill you wrote whose contract still lists `check_after_last_edit` keeps the rest of its contract. (#4439)
- The ship policy guide now explains how to merge any pull request with `gstack-ship-queue land`, even in a repository with one maintainer. (#4377)
- /ship's test step runs a large unit suite on Linux. (#4426)
- Every skill that loads the shared preamble, on every host, now says never to print a whole environment, git config or process table. (#4382)
- Pull requests from /ship now open with at most three plain lines, and every merge lands as one commit titled with the PR title. (#4438)
- `gstack-pr-wait`, `gstack-ci-gate` and the release cut no longer call a commit green while a check on it failed. (#4257)
- `gstack-runaway` now says what a thread's cold and empty wakes cost, not just how many there were (GST-2770). (#4385)
- The optional Stop hook no longer checks for a test run after the last edit, and /verify asks for it in plain words instead. (#4428)
- /ship's test step says when it ran out of time, and you can set its budget. (#4429)
- The remote test command no longer puts every file on one command line. (#4429)
- The stable release no longer waits on the nightly Windows tests (GST-2815). (#4400)
- Ubicloud fan-outs can no longer overshoot the vCPU limit, and owners no longer delete each other's VMs. (#4430)
- hono is pinned to 4.13.16, which fixes five advisories the weekly OSV scan found in 4.13.13 (gstack#4406). (#4408)
- `package.json` overrides (#4408)
- `test/dependency-security.test.ts`: the lockfile floor check now covers `hono >= 4.13.16`. (#4408)
- The nightly Windows test run finishes again. (#4409)
- A short write-up checklist in ETHOS.md. (#4381)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant