Skip to content

v1.94.9.0 feat(pregate): owner-granted flaky exception for named cases in a red run (GST-1096) - #3911

Merged
capy-ai[bot] merged 5 commits into
mainfrom
gsta-258-b-flaky-exception
Nov 3, 2026
Merged

capy-ai[bot] merged 5 commits into
mainfrom
gsta-258-b-flaky-exception

Conversation

@garrytan

@garrytan garrytan commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner
{
  "schema_version": 1,
  "pr": 3911,
  "head": "4340ed4ca4d984db2ee83f00caba434ad2729020",
  "version": "1.94.4.0",
  "base": "main",
  "gate": {
    "focused": {
      "pass": 319,
      "fail": 0,
      "skip": 0
    },
    "ci-carry": {
      "pass": 34598,
      "fail": 0,
      "skip": 0
    }
  },
  "session_kind": "unattended",
  "head_tree": "1f243ef417aedbb4accec66abe11832203bf7faf",
  "product_patch_id": "216450f77d4b3e4078083c1af04384d0cc22b818",
  "gstack_version": "1.94.4.0",
  "review_depth": {
    "depth": "full",
    "class": "gate",
    "runtime_lines": 638,
    "rules": "18dcd562316f56ed",
    "base": "5f705f367ae396aff143080a8eafd9951f32f70c",
    "registry": "5f705f367ae396aff143080a8eafd9951f32f70c",
    "gate_scan": {
      "tagged": 0,
      "head": "4340ed4ca4d984db2ee83f00caba434ad2729020"
    },
    "reasons": [
      "gate-class",
      "risk:security",
      "runtime=638"
    ],
    "risk": [
      "security"
    ]
  },
  "pregate": "requires-remote cleared @1f243ef417ae (7 checks, current for 4340ed4ca4d9)",
  "requires_remote": [
    "free-tests.yml/macos-named-regressions: https://github.com/garrytan/gstack/actions/runs/41579867879/attempts/1",
    "pregate.json/remote_gate: remote-gate tree=1f243ef417ae tests=34598 skipped=73 runs=41579867879/1 profiles=ci-reuse/free-tests heads=4340ed4ca4d9 verified=2026-11-03T04:50:43.663Z"
  ],
  "issues": {
    "fixes": [],
    "refs": [],
    "flakes": []
  },
  "docs": {
    "status": "current",
    "scope": "docs/pregate.md, changelog.d fragment, docs/ship-policy.md, docs/troubleshooting.md (docs/, README.md, AGENTS.md roots)"
  },
  "review_loop": {
    "passes": 1,
    "blocking_found": 3,
    "deferred": 0,
    "noted": 1,
    "fixed": 0,
    "scope": "frozen",
    "fix_checks": 2,
    "rounds": 3,
    "unrecorded": 3,
    "ledgers_read": 2,
    "dispatches": {
      "specialist": 3,
      "red-team": 1
    }
  },
  "review_loop_exception": {
    "accepted_by": "GStack Minder",
    "date": "2026-11-03",
    "via": "standing grant, GSTA-258 builds, 2026-11-03",
    "reason": "rule 19 merge of main 5f705f36 into #3911; one scoped fix-check on the merge hunks only",
    "fix_check": {
      "head": "4340ed4ca4d984db2ee83f00caba434ad2729020",
      "findings": 0
    }
  }
}

What changed

gstack-pregate clear now takes --flaky-exception with --pr and --grant, so an owner can let named flaky cases in an untouched test file clear the pregate. (The image render failed in this environment, so the before/after is text only.)

Before/after as text: gstack-pregate clear now takes --flaky-exception with --pr and --grant, so an owner can let named flaky cases in an untouched test file clear the pregate
--- Before · before.txt · captured output
+++ After · after.txt · captured output
@@ ⋯ 2 unchanged ⋯ @@
                   verifies GitHub Actions runs against the base branch's run_profiles instead (docs/pregate.md#run-evidence):
                   one run of the profile named after the lane, or for pregate.json/remote_gate one run per
                   remote_gate_runs profile covering the gate's inventory (needs --allow-repo-commands); where a lane has a
+                  profile, --run-url is verified the same way. --flaky-exception "<test file>::<case>" (at most 3) --pr <n>
+                  --grant <approval id> takes such runs with failures when every failure is named from the run's shard
+                  results and logs, outside the PR's diff and import closure, and an owner's gstack-approvals row binds
+                  them; without --grant it prints the row to record (pregate only; docs/pregate.md#flaky-exception)
-                  profile, --run-url is verified the same way
 

Why

GSTA-258 design piece 6 (GST-1096): "a timing flake in a test file the PR never touched keeps the pregate red, with no way through but another paid run." The owner can now accept up to three named failing cases from one red Free Tests or Windows Free Tests run for one PR head. Every required CI check still needs its own green run at the head, so the exception never waives CI.

Blast radius

The change adds a new path to gstack-pregate clear, a new pass-with-exception verdict that verify, ready and the ship receipt read, and a flaky_exception receipt field. Every existing clear path is untouched, because the exception code runs only when --flaky-exception is passed. lib/ship-release.ts:373 matches /^(pass|warn|requires-remote cleared)/, so finish accepts the new verdict (file:line read). At this head, CI carry accepted Free Tests run 41579867879 at tree 1f243ef4 (34,598 tests), which cleared the remote gate (script ran).

Approvals

No approvals recorded for branch gsta-258-b-flaky-exception.

Summary

A red run whose only failures are named flaky cases can now clear a requires-remote pregate obligation once the owner grants it. The grant is a gstack-approvals row whose scope hashes the repository, PR, head, tree, lane, run attempt and case list, so it cannot move to another head or PR. The run's own shard results and logs must name every failure (B8: a red run uploads no tree.json, so the evidence comes from the strict shard artifacts). A case in the PR's diff or import closure, a paid test file or a security test refuses. In the review round, the printed grant command started quoting case names for the shell; ready and the receipt refuse an exception granted for another PR or head; red runs are read again after the inventory and grant checks; the security-test match is wider; and the clear-only code loads scripts/ and test/ modules lazily, so a bin+lib install still loads the receipt bin.

This change never touches the Ubicloud --receipt gate. That gate's output is text the lane produces itself, so a grant has nothing independent to bind to. Build A's CI carry handles flakes in the remote pregate.

Test Coverage

Coverage audit (Step 7, audit only, at the first review tree fe9fc759): 30 new code paths, 17 covered (57% any test, 57% value-weighted), 13 gaps. The fix round then added tests for the biggest gaps: the run provenance refusals (other_repository, fork_run, workflow_mismatch, event_mismatch, run_in_progress, run_stale), a red run that moves after the inventory runs, shell quoting of case names, two more security-test names, and ready refusing an exception bound to another PR.
Tests: 1667 test files before, 1667 after (the new file came in the first commit; the fixes extended it).
Gaps still open: the real ghArtifactDeps and approvalGrants boundaries are always replaced by fakes in tests, so the real GitHub artifact format is unproven; the receipt's other-head refusal has no test of its own.
Tests ran: focused 408

Pre-Landing Review

Review depth: full (gate class, security risk). Pass 1 (tree fe9fc759) ran a specialist batch (testing, security, state machines), a red team, the native adversarial reviewer and an outside voice (GPT-6.1 Sol). Together they returned 13 findings. Three were marked blocking by the outside voice: shell injection in the printed grant command, a grant reused across heads with the same tree, and red runs not read again after the inventory. All three are fixed in a96e76d, along with the security-test regex gap, an honest note in the docs on the owner check, and the import-closure limit. Pass 2 (range 549613e..a96e76d, specialist plus red team) found no blocking issues and one INFORMATIONAL testing note.
REVIEW_LOOP: passes=1 blocking_found=3 deferred=0 noted=1 fixed=0 scope=frozen unrecorded=3 ledgers_read=2 fix_checks=2 dispatches=4
Review-loop exception: accepted by GStack Minder 2026-11-03 via standing grant, GSTA-258 builds, 2026-11-03: rule 19 merge of main 5f705f3 into #3911; one scoped fix-check on the merge hunks only (fix-check at 4340ed4, 0 findings)
Gate edits: none detected (GATE_SCAN tagged=0)
Outside review: GPT-6.1 Sol, request changes on pass 1; every finding it raised is fixed.

Deferred findings

Review notes

After the rule 19 merge of main at 5f705f3 (one conflict, the imports in lib/pregate/index.ts, kept both; #3901's carryGate taken as main wrote it), one scoped fix-check read the merge hunks and returned NO FINDINGS. Its review-log row was written twice by a repeated command, so the loop line shows fix_checks=2 for one fix-check.

The REVIEW_LOOP line shows passes=1 and fixed=0 because pass 1 was run without first taking a review-log start token, so its row could not be written honestly. Its briefs, outputs and judgment are in that pass's dispatch ledger. Pass 2's row is bound to tree 42cea903.

Exploratory QA

No browser or UI surface. The behavior is covered by the fixture tests and by the remote pregate at this tree.

Design Review

No frontend files changed — design review skipped.

Eval Results

No prompt-related files changed — evals skipped.

Review bot comments

No review bot comments.

Plan Completion

Plan completion audit: not run (no plan is bound to this branch and no docs/designs/ file matches). The design is GSTA-258's ship-90-90 design, piece 6, with fixes B8 (red runs have no tree.json; evidence comes from the shard artifacts) and B9 (the exception is pregate only, and required CI still needs green). B10 is covered by the head-and-PR binding.

Verification Results

No Step 8.1 obligations were bound.

TODOS

No TODO items completed in this PR.

Documentation

Status: current. No documentation edits were needed.

Scope: base 14d7414 through head a96e76d plus the unstaged changelog.d fragment edit. The 12 selected changed files were bin/gstack-pregate, bin/gstack-ship-receipt, the changelog fragment, docs/pregate.md, lib/headless-artifacts.ts, lib/pregate/flaky-exception.ts, lib/pregate/index.ts, lib/pregate/run-check.ts, lib/run-evidence.ts, lib/ship-queue.ts, lib/ship-receipt.ts and test/pregate-flaky-exception.test.ts.

Documentation health:

  • docs/pregate.md: current. The new Owner-granted flaky exception section matches the code (reason codes, the 3-case cap, the security-test pattern, grant binding, the pass-with-exception verdict, and the ready and receipt output).
  • changelog.d/gsta-258-b-flaky-exception.md: current. The unstaged wording tweak is fine and was left as is.
  • docs/ship-policy.md and docs/troubleshooting.md: current. They do not mention the new flags, but nothing in them is contradicted.
  • README.md, AGENTS.md: not affected by this change.

Coverage debt:

  • --flaky-exception, --pr and --grant on gstack-pregate clear: reference and how-to are in docs/pregate.md (plus the --help text); explanation is the "Why not base_red" subsection. There is no tutorial. Not listed in docs/troubleshooting.md or docs/ship-policy.md, which is a minor gap.
  • flaky_exception field in the ship receipt and the FLAKY_EXCEPTION: READY line: documented in docs/pregate.md only.
  • Diagram drift: none; the changed docs have no architecture diagrams.

Test plan

  • focused tests after the main merge (17 files): 319 pass, 0 fail; bun run typecheck clean; bun run typecheck:test 814 known, none new
  • CI carry at 4340ed4: CI_CARRY: cleared pregate.json/remote_gate remote-gate tree=1f243ef417ae tests=34598 skipped=73 runs=41579867879/1
  • CI at 4340ed4: every check green
  • Before the merge: remote pregate on Ubicloud at tree 3e36b4bf ran 9785 tests with 0 failed

garrytan and others added 3 commits November 3, 2026 01:41
… run (GST-1096)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…he grant command, recheck red runs (review fixes)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

@capy-ai
capy-ai Bot marked this pull request as ready for review November 3, 2026 02:31
…-b-flaky-exception

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

# Conflicts:
#	lib/pregate/index.ts
@github-actions github-actions Bot changed the title feat(pregate): owner-granted flaky exception for named cases in a red run (GST-1096) v1.94.9.0 feat(pregate): owner-granted flaky exception for named cases in a red run (GST-1096) Nov 3, 2026
@capy-ai
capy-ai Bot merged commit f33fd30 into main Nov 3, 2026
46 checks passed
garrytan added a commit that referenced this pull request Nov 3, 2026
lib/ship-receipt.ts KEY_ORDER keeps flaky_exception (#3911) and no_stop.
bin/gstack-ship-receipt: main's review_exclude help and digest refusal text plus
--no-stop help; the J1 re-check block before one planCarry call carrying both
stackParent (#3928) and recheck; the recheck risk-class check after the digest check.
test/review-packets-removed.test.ts keeps both NO_LOSS_REPLAYS keys (GST-940 and
GST-508+GST-464+GST-967+GSTA-258-L4).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
garrytan added a commit that referenced this pull request Nov 3, 2026
docs/pregate.md keeps the ci-reuse-pr subsection and main's Owner-granted flaky
exception section (#3911), in that order.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant