Skip to content

Stop a merge check from passing a test that was weakened after review (GST-2204) - #4248

Closed
garrytan wants to merge 7 commits into
mainfrom
receipt-carry-refuses-weakened-tests
Closed

garrytan wants to merge 7 commits into
mainfrom
receipt-carry-refuses-weakened-tests

Conversation

@garrytan

@garrytan garrytan commented Nov 9, 2026

Copy link
Copy Markdown
Owner

A receipt carry no longer lets a test that was weakened after review merge without anyone reading it. This closes a hole in the merge checks (GSTA-461 audit item 6).

What changed

  • gstack-ship-receipt carry used to treat every edit under test/ as harmless. After /ship, a lane could change expect(a).toBe(1) to expect(typeof a).toBe('number'), carry the receipt as a "test-only change", and land with CI green and no reviewer. Now a carry whose test diff removes or changes an assertion line (an expect, assert or matcher, or a test(...)/it(...) line) refuses with RECEIPT_CARRY_TEST_WEAKENED. The refusal names each hunk as path:line and writes a packet for one scoped fix-check. The carry goes through once --test-review <record.json> holds a clean verdict bound to that head and the packet's sha256.
  • New tests and new assertions still carry as a test-only change. An assertion the base changed on its own, merged in, still carries as a base merge. A skipped test or a deleted test file still refuses through the review-depth check, which runs first.
  • test/land-merged-checks.txt (the list land runs on a moved base) and the touchfile maps under test/helpers/ now count as product for a carry. Emptying or editing them is RECEIPT_CARRY_PRODUCT_CHANGED, not a test-only change.
  • New troubleshooting entry (#receipt-carry-test-weakened), a row in docs/ship-policy.md's carry table, a gstack-ship-preflight route line and a changelog fragment.

Tests

test/ship-receipt-carry-weakened-test.test.ts was written first from the audit repro. On main, the repro carried and the gate-path and parser cases failed (6 of 9 failed). On this branch all 9 pass. The negative controls are a pure test addition, a base-side assertion change and test.skip.

Agent notes
  • Detection is in lib/ship-carry-tests.ts. It reads removed lines in the zero-context diff from the reviewed head to HEAD, minus lines the base removed itself, plus lines the branch's current patch drops that its reviewed patch didn't (an assertion a base merge brought in that the branch then removed).
  • The new refusal is placed after the depth recheck, so test.skip and deleted tests keep their stronger refusal (/ship).
  • Gate test paths are compared path by path (comparePath) because the product patch-id pathspec can't re-include a path under an excluded test/** glob. planCarry now refuses whenever changedProduct is non-empty, even when the patch-ids match. This changes nothing for existing callers, which filled changedProduct only when the patch-ids differed.
  • Known limits: an added early return or a new mock is an added line and still carries (the gate scan and CI cover those). Retargeted stacked children skip the gate-path comparison.

Open in Capy

@trunk-io

trunk-io Bot commented Nov 9, 2026

Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@capy-ai capy-ai Bot changed the title Make a receipt carry refuse a test weakened after review (GST-2204) Stop a merge check from passing a test that was weakened after review (GST-2204) Nov 9, 2026
garrytan and others added 4 commits November 9, 2026 23:45
A carry treated every edit under test/ as harmless, so a loosened assertion rode a test-only carry to land unread. A carry whose test diff removes or changes assertion lines now refuses with RECEIPT_CARRY_TEST_WEAKENED naming the hunks and carries only after one scoped fix-check bound to its packet. The land merged-checks list and the touchfile maps count as product for a carry.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Review-Finding: bin/gstack-doctor-components.sh:340:enum-completeness reader=core
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…spawn the receipt script

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@garrytan
garrytan force-pushed the receipt-carry-refuses-weakened-tests branch from c48c072 to 7d6be7a Compare November 9, 2026 23:50
garrytan and others added 3 commits November 10, 2026 00:26
…t eval policy, budgets and ratchet allowlists as gate files

A base removal used to cancel a branch removal whenever the assertion text matched, so the base dropping an expect from one test could hide the branch dropping the same expect from another. Removals now match only within the same test at the same place. The paid-eval policy and budget data and the ratchet allowlists join the land list and touchfile maps as gate files a test-only carry cannot hold.

Review-Finding: v1/outside-voice#3 reader=outside-voice
Review-Finding: v1/adversarial-native#3 reader=adversarial-native
Review-Finding: v1/targeted#2 reader=targeted
Review-Finding: p1/red-team#1 reader=red-team
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
The gate-file check was skipped when a stacked child is retargeted after its parent lands, so the child could empty the land list after review and still carry. On that route each gate file must now match the reviewed head byte for byte.

Review-Finding: v1/outside-voice#1 reader=outside-voice
Review-Finding: v1/adversarial-native#4 reader=adversarial-native
Review-Finding: v1/targeted#1 reader=targeted
Review-Finding: p1/specialist-batched#1 reader=security
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
The check reads removed or rewritten assertion lines only. The docs and changelog now name what still carries as test-only: a value changed inside a multi-line matcher, an added early return, and golden or snapshot edits.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@capy-ai

capy-ai Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Closing this for now. The check that stops a test weakened after review from merging without a reviewer is parked until the design for owner rulings on review findings (GST-2273) lands. The branch stays at 1556ce2 and keeps the review fix round, so the work can be picked up from there.

@capy-ai capy-ai Bot closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant