Skip to content

v1.91.302.0 fix: repair three flaky gate cases from the 2026-10-28 flaky pass (index flags, exploratory QA, cookie auth) - #3553

Merged
capy-ai[bot] merged 11 commits into
mainfrom
flaky-2026-10-28-s6
Oct 29, 2026
Merged

capy-ai[bot] merged 11 commits into
mainfrom
flaky-2026-10-28-s6

Conversation

@garrytan

@garrytan garrytan commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner
{
  "schema_version": 1,
  "pr": 3553,
  "head": "dd76b43882e41d3cb12aa0bc856f86a3c9fc26b0",
  "version": "1.91.295.0",
  "base": "main",
  "gate": {
    "pregate.json/remote_gate": {
      "pass": 19180,
      "fail": 0,
      "skip": 0
    }
  },
  "session_kind": "unattended",
  "pregate": "requires-remote cleared @039b6ebdc4ca (1 checks, current for dd76b43882e4)",
  "requires_remote": [
    "free-tests.yml/macos-named-regressions: https://github.com/garrytan/gstack/actions/runs/40746083189",
    "windows-free-tests.yml/windows-free-shard: https://github.com/garrytan/gstack/actions/runs/40746083344",
    "pregate.json/remote_gate: remote-gate tree=039b6ebdc4ca tests=19180"
  ],
  "issues": {
    "fixes": [],
    "refs": [
      2964
    ],
    "flakes": []
  },
  "docs": {
    "status": "current",
    "scope": "AGENTS.md, CLAUDE.md, CONTRIBUTING.md, README.md, docs/ (128 docs) against 61e201e..dd76b43"
  },
  "review_loop": {
    "passes": 1,
    "blocking_found": 0,
    "deferred": 0,
    "noted": 0,
    "fixed": 0,
    "scope": "frozen",
    "dispatches": {
      "specialist": 1
    }
  }
}

What changed

Ship's exploratory-QA fixture now admits Step 9's gstack-review-plan call at the installed path and nothing else; both refused paid-screen captures replay clean

Ship's exploratory-QA fixture now admits Step 9's gstack-review-plan call at the installed path and nothing else, so both refused captures from paid screen 40707142223 replay clean.

Before/after as text: Ship's exploratory-QA fixture now admits Step 9's gstack-review-plan call at the installed path and nothing else; both refused paid-screen captures replay clean
--- Before · before.txt · captured output
+++ After · after.txt · captured output
+$ bun test test/qa-caller-review-plan.test.ts   # flaky-2026-10-28-s6
+(pass) captured paid screen slice-10: every Bash call the actor made is inside the declared interface
+(pass) captured paid screen rerun-1: every Bash call the actor made is inside the declared interface
+(pass) the helper forms Step 9 names are admitted at the exact installed path, and nothing else is
+(pass) ship-exploratory-small-cli: every reviewer the real plan dispatches has a declared route in the transport
+(pass) ship-exploratory-unavailable: every reviewer the real plan dispatches has a declared route in the transport
+(pass) ship-exploratory-plan-checks: every reviewer the real plan dispatches has a declared route in the transport
+(pass) ship-exploratory-late-input: every reviewer the real plan dispatches has a declared route in the transport
-$ bun test test/qa-caller-review-plan.test.ts   # fix stashed (base transport)
-(fail) captured paid screen slice-10: every Bash call the actor made is inside the declared interface
-(fail) captured paid screen rerun-1: every Bash call the actor made is inside the declared interface
-(fail) the helper forms Step 9 names are admitted at the exact installed path, and nothing else is
-(fail) ship-exploratory-small-cli: every reviewer the real plan dispatches has a declared route in the transport
-(fail) ship-exploratory-unavailable: every reviewer the real plan dispatches has a declared route in the transport
-(fail) ship-exploratory-plan-checks: every reviewer the real plan dispatches has a declared route in the transport
-(fail) ship-exploratory-late-input: every reviewer the real plan dispatches has a declared route in the transport
 (pass) review keeps its own Step 4.8 transport and is not told about ship Step 11
 (pass) captured late-input receipt: pass with remaining work is still a false green
+ 9 pass
+ 0 fail
- 2 pass
- 7 fail
 

Shared-code reuse step 1 now says evidence gathering is read-only and names the command that cleared the assume-unchanged flag

Shared-code reuse step 1 in /review and /ship now says evidence gathering is read-only and names git update-index --really-refresh, the command that cleared the assume-unchanged flag in gate 40619718717.

Before/after as text: Evidence gathering is now read-only and names git update-index --really-refresh, the command that cleared the assume-unchanged flag in gate 40619718717
--- Before · ship/sections/shared-code-reuse.md at 61e201e · source text
+++ After · ship/sections/shared-code-reuse.md on this branch · source text
@@ ⋯ 1 unchanged ⋯ @@
 <!-- Regenerate: bun run gen:skill-docs -->
 **Reuse a skipped shared-code advisory only with complete structural evidence:**
 
+1. **Read the evidence.** Read all supporting callers and the helper target
+   read-only: `git update-index`, even `--really-refresh`, clears the index flags
+   the checker reads. Establish first-party authored provenance and whether this
+   extraction is worthwhile; the checker cannot. Retain `evidence_paths`/`helper_target`.
-1. **Read the evidence.** Read all supporting callers and the helper destination.
-   Establish first-party authored provenance and whether the current extraction
-   is worthwhile; the checker cannot decide that. Retain `evidence_paths`/`helper_target`.
 2. **Run the checker.** From the repository root, pass the current finding as
+   literal JSON on stdin. Replace REVIEW_START with this pass's token
-   literal JSON on stdin. Replace REVIEW_START with this pass's captured token
    and the example paths/symbol with actual evidence. Keep the quoted delimiter.
 
 ```bash
@@ ⋯ 2 unchanged ⋯ @@
 GSTACK_SHARED_LIBS_REUSE_JSON
 ```
 
+3. **Act on its result.** Only `reusable: true` permits suppression.
-3. **Act on its result.** Read the JSON. Only `reusable: true` permits suppression.
    False, command failure or unreadable output requires fresh source review and a
    new decision, never suppression. Do not supply your own snapshot, prior record or coverage.
+4. **Persist through the logger.** The logger recomputes coverage; never supply
+   proof. Real defects retain normal Fix-First handling.
-4. **Persist through the logger.** The logger recomputes final coverage; never
-   supply proof yourself. Real defects retain normal Fix-First handling independently.
 
+**What a reusable result proves (never reconstruct these checks):**
+- Identity: `sharedLibsFingerprint` plus the repo, raw branch and current snapshot.
-**What a reusable result proves (do not reconstruct these checks yourself):**
-- Identity: `sharedLibsFingerprint` plus the actual repo, raw branch and current snapshot.
   The checker reads REVIEW_START without consuming/replacing it. Sanitized branch names are not identity.
 - Prior decision: completed/converged review, verified binding, explicit Skip and
+  logger-versioned `snapshot_covered_paths`; unversioned coverage needs a fresh decision.
-  logger-versioned `snapshot_covered_paths`; older unversioned coverage needs a fresh decision.
 - Source: `canReuseSharedLibsAdvisory` requires every supporting path's raw file
   byte-for-byte with its blob. Exclude assume-unchanged, skip-worktree and sparse index
   entries; symlinks/ancestors, submodules, ignored/outside or unreadable files;

Why

The 2026-10-28 daily flaky pass (Garry: "we should have a flaky test fix pass every day that tries to fix flaky tests") found three gate cases that failed for reasons we could see in their captures. A reviewer refreshed the index and erased the flag it was checking, ship's exploratory-QA fixture refused the review-plan command that #3527 added to Step 9, and the real-browser cookie-auth checks used a 400 ms limit that no product path uses. Each case now fails only when the product is wrong.

Blast radius

The prompt edit changes the rendered shared-code-reuse.md in review/ and ship/ and nothing else. Both renders keep their exact byte size (2342), so no size pin or context ledger moves; the generated sections and gen:skill-docs freshness check were run. The transport and cookie-auth edits are test-only: reviewPlanCommandAllowed is a non-exported helper in test/helpers/qa-callers-fixture.ts, and browse/test/cookie-auth-verification.test.ts changes only the options its real-Chromium describe passes. The Windows curation entry removes a test that is new on this branch, so no test that ran on Windows before stops running there (file:line read, scripts/test-free-shards.ts --windows-only --list ran).

Approvals

Date Asked Exact words Outcome Covers Still pending
2026-10-28 cookie-auth real-browser cases use the product default timeout “approved. The real-browser positive and rejection cases use the product default (no `timeoutMs`, so the 15 s cap applies). The explicit 100 ms timeout tests stay as the timeout proof. Record it as a gate edit with the provenance (a84b0b5, #2964, a fixture value that no product…” (GStack Minder) approved browse/test/cookie-auth-verification.test.ts real-Chromium positive and rejection cases none
2026-10-28 cookie-auth identity_* rejection cases keep the 400 ms options (narrowing of the default-timeout ruling) “Accept. The three identity_* rejection cases keep 400 ms, because their outcome only arrives when the deadline fires and the product default would trip bun's 5 s test timeout.” (GStack Minder) approved browse/test/cookie-auth-verification.test.ts identity_mismatch, identity_missing and identity_ambiguous rejection cases none

Source: gstack-approvals record for flaky-2026-10-28-s6 (display only; not an approval citation).

Summary

Three flaky gate cases are repaired, each diagnosed from its captured run first:

  • shared-libs-review-index-flags (prompt edit). In train gate 40619718717 the failing trial ran git update-index --really-refresh while gathering evidence, which cleared the assume-unchanged (h) flag the checker reads, so its prerequisites went stale. The passing rerun never ran it. Step 1 of the shared-code reuse section, rendered from scripts/resolvers/review-scope.ts, now says evidence gathering is read-only and names the command.
  • ship-exploratory-late-input and ship-exploratory-plan-checks (gate edit). Since feat(ship): /ship and /autoplan size their review to the diff (GSTA-131) #3527, ship Step 9 runs gstack-review-plan --base main. The qa-callers fixture refused it (paid screen 40707142223, both plan-checks trials) and never said that the plan's adversarial-native and outside-voice lines belong to Step 11, outside the fixture's Steps 8–9 excerpt. In gate 40664336141, late-input reported pass while listing that Step 11 review pass as remaining work. Both cases share this root cause. Once the plan ran, it dispatched the state-machines specialist, and review-army tells the parent to write the merge-base sha into that specialist's diff command. The fixture refused that child's git diff <merge-base sha>, which failed both ship-exploratory-unavailable trials in this PR's first screen. The transport now admits the fixture's own merge base, bare or in review-army's DIFF_BASE=<sha> && git diff "$DIFF_BASE" form, and the prompt declares it.
  • cookie-auth real-Chromium cases (gate edit, approved by GStack Minder). The positive case and the login, HTTP and target rejection cases now use the product default timeout. The identity mismatch, missing and ambiguous cases keep 400 ms, as GStack Minder accepted, and a comment beside them cites the deadline loop in browse/src/cookie-auth-verification.ts.

In the first screen, shared-libs-review-revalidation failed once and passed its rerun. That was a flake, not this edit. In the filtered scenario, the actor ran echo ===ROUTE under zsh, where =ROUTE is command-path expansion. The call exited 1 before retry-worker.ts was read, and the rerun quoted the separator. Both runs read every evidence path as step 1 says, and neither touched the index.

A Windows curation entry follows from the first fix: test/shared-libs-index-flags-readonly.test.ts launches bin/gstack-review-log through its fixture, which windows-latest cannot run (Windows Free Tests run 40727825859, shard 5).

Test Coverage

Two new free test files cover the repairs, and each fails without its fix:

  • test/qa-caller-review-plan.test.ts: 12 tests. It replays both captures from that paid screen and both ship-exploratory-unavailable captures from screen 40737741454, checks the admitted helper forms with negative controls (pipes, other arguments, another binary, no runtime), and runs the real plan for all four ship cases. Regression proof: 7 tests fail without the review-plan admission, and 3 more fail without the merge-base admission. With both, all 12 tests pass.
  • test/shared-libs-index-flags-readonly.test.ts: 26 tests. It replays the captured refresh against the assume-unchanged fixture with a read-only control, and checks the rendered rule on every host. Regression proof: 2 tests pass and 24 tests fail against main's resolver, 26 tests pass on this branch.
  • cookie-auth: all 17 runs pass at 0, 8, 16 and 24 busy loops; with the shared 400 ms options the identity cases resolve only at the 15 s cap and hit bun's 5 s test timeout.

Coverage percentages were not recomputed for this body.

Pre-Landing Review

Pre-Landing Review: no blocking issues. Informational findings are listed under Review notes. One specialist pass, a native adversarial pass on the gate and fixture code, and a cross-family outside voice. Three fix-checks covered the later commits. 49f780c was clean, and 60f0780 had one informational item, listed under Review notes. The screen repair (ad7e2ef) went to a reviewer with an adversarial brief, who found no over-admission. That reviewer flagged one informational prompt inconsistency, and dd76b43 fixes it.

REVIEW_LOOP: passes=1 blocking_found=0 deferred=0 noted=0 fixed=0 scope=frozen dispatches=1

Gate edits: 18 listed, 9 read, 3 findings — 3 kept (reasons below), 0 restored, 0 open

  • kept: qa-callers command interface, test/helpers/qa-callers-fixture.ts. The fixture must admit the review-plan helper that ship Step 9 runs (feat(ship): /ship and /autoplan size their review to the diff (GSTA-131) #3527); admission is limited to the installed path and literal argument forms, with negative controls.
  • kept: merge-base diff admission, test/helpers/qa-callers-fixture.ts. review-army's specialist dispatch writes the merge-base sha into the child's diff command. Only a prefix of the fixture's own git merge-base origin/main HEAD of at least 7 characters is admitted, with negative controls.
  • kept: cookie-auth timeout options, browse/test/cookie-auth-verification.test.ts. The shared timeoutMs: 400 came from a84b0b5 (v1.90.0.0 feat: make browser cookie imports explicit and safe #2964) as a fixture value no product path uses (--verify-auth callers run under MAX_TIMEOUT_MS 15000). Approved by GStack Minder. The identity_* cases keep 400 ms under GStack Minder's second ruling (see Approvals).

Outside review: openai/gpt-6.1-sol (Capy subagent), verdict clean, no findings.

Review notes

  • The identity mismatch, missing and ambiguous rejection cases keep the 400 ms options. They resolve only at the deadline, so the 15 s default would exceed bun's 5 s test timeout (0 of 17 runs). GStack Minder accepted this narrowing.
  • The review-plan admission is not scoped to the ship caller; review's own transport is separate and is checked by a test.
  • --fix-check-prompt operands accept any literal path, including absolute and .. paths. The fixture only checks the command shape, and the helper reads the files.
  • The ship prompt says the native reviewer is still required, and also that Step 11's voices are outside the excerpt. The two lines can read as conflicting.
  • --pass $PASS (a shell variable) is refused; only literal numbers are admitted.
  • In scripts/lib/windows-curation.ts, the new entry sits between the credential-rejection comment and its entry. A swap was reverted because it would have changed an input the documentation audit had hashed.

Exploratory QA

Not run: the diff has no user-facing surface. The repaired cases are themselves exploratory-QA eval fixtures, and their paid runs are under Eval Results.

Design Review

No frontend files changed — design review skipped.

Eval Results

Focused paid runs at 24bf7e5, one trial each, all pass, $5.50 of the $10 lane cap: shared-libs-review-index-flags $1.00, ship-exploratory-late-input $1.05, ship-exploratory-plan-checks $0.86, review-exploratory-small-cli $0.76, ship-exploratory-small-cli $0.90, ship-exploratory-unavailable $0.93. Both the late-input and plan-checks transcripts run gstack-review-plan --base main. Commits after 24bf7e5 change tests, a curation list and docs only, so the prompts these cases read are byte-identical. These results are reused evidence for the three cases this PR's later commits do not touch.

First screen (evals run 40737741454, at 4cd0777): RED. ship-exploratory-unavailable failed both trials on the specialist diff that this PR now admits. shared-libs-review-revalidation was flaky ($1.76 rerun), and the cause is described under Summary. After the repair, a focused run of ship-exploratory-unavailable at dd76b43 passed ($0.82), and its specialist child ran DIFF_BASE=<sha> && git diff "$DIFF_BASE". A run at ad7e2ef was cancelled at about 65 s for the prompt fix, and its cost was not reported. Lane focused spend: $6.32 plus that partial run, against the $10 cap.

Greptile Review

No Greptile comments.

Issues

Refs #2964

Verification Results

Each repair was checked by its failing-first test and its focused paid case: all three repairs PASS; none failed, blocked or not run. Windows Free Tests on the branch are listed under Test plan.

TODOS

No TODO items completed in this PR.

Documentation

Status: current. No documentation edits were needed for ad7e2ef and dd76b43.

Scope: the selected diff from 61e201e to dd76b43, focused on the qa-callers changes since 80955d9 (test/helpers/qa-callers-fixture.ts, the new fixture, the new tests, a cookie test comment and CHANGELOG). Nothing was staged, unstaged or untracked.

Documentation health:

  • docs/TESTING_INTERNALS.md: Current
  • docs/TEST_PORTFOLIO.md: Current (names the cases only)
  • docs/review-policy.md: Current
  • docs/qa-without-a-webpage.md: Current (no qa-callers references)
  • docs/evals/detector-inventory.md: Current

Coverage debt: the caller command interface has no reference doc; the fixture prompt is the only description of its Git forms. This is an existing gap. Diagram drift: none.

Test plan

  • gstack-pregate --stage tests --remote --spend-cap 2 --timeout 2700 --allow-repo-commands at dd76b43: 19180 tests, 0 failed, $0.07; macOS named regressions and Windows lanes cleared from this PR's Free Tests and Windows Free Tests runs at dd76b43, both green
  • Windows Free Tests at 4cd0777 (run 40733429080, dispatched on the branch): six shards green, browse/test/cookie-auth-verification.test.ts passing there
  • Local guards: bun run typecheck 0 errors, bun run typecheck:test no new diagnostics; qa-caller-review-plan, qa-exploratory-callers, qa-caller-authority, qa-caller-report-observer, qa-checkpoint-evidence, qa-evidence-producer, free-fixtures, spawnsync tripwire, test-of-test and module-size ratchets, harness-version, cookie-auth-verification all green after the fixture was tracked
  • Failing-first: bun test test/qa-caller-review-plan.test.ts 3 tests fail without the merge-base admission and 12 tests pass with it; 7 tests fail without the review-plan admission; bun test test/shared-libs-index-flags-readonly.test.ts 24 tests fail against main's resolver, 26 tests pass on the branch
  • cookie-auth under load (0, 8, 16 and 24 busy loops): all 17 runs pass
  • Focused paid cases: all six pass at 24bf7e5 ($5.50), ship-exploratory-unavailable passes at dd76b43 ($0.82)

🤖 Generated with Claude Code

garrytan and others added 8 commits October 28, 2026 16:35
… flaky, gate 40619718717)

The failing assume-unchanged trial ran git update-index --really-refresh while gathering evidence, which cleared the flag the checker reads. Step 1 now says the read is read-only and names that command; nearby prose tightened so the section stays byte-identical in size (no pin or ledger move).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…s Step 11 voices (late-input 40664336141, plan-checks 40707142223)

Gate edit: the caller interface admits the gstack-review-plan forms Step 9 names at the exact installed path; the ship prompt says the plan's adversarial-native and outside-voice lines are Step 11's, outside the Steps 8-9 excerpt. Free replays of both 40707142223 captures and the real plan for all four ship cases.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…ection cases use the product default timeout

Gate edit approved by GStack Minder. The shared 400 ms came from a84b0b5 (#2964) as a fixture value no product path uses; --verify-auth callers run under MAX_TIMEOUT_MS 15000. Identity mismatch/missing/ambiguous cases resolve only at the deadline, so they keep the 400 ms bound; explicit timeout tests are unchanged.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
… host (test-of-test ratchet)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Its fixture launches bin/gstack-review-log, a bash script, directly; windows-latest
reports it not found in PATH (Windows Free Tests run 40727825859, shard 5).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…s own entry"

The swap changed an input the s6-docs-2 documentation audit hashed, and /ship
allows no third audit in one invocation. The misplaced comment is listed as a
review note instead.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

E2E Evals: ✅ PASS (flaky: 2 case(s) passed on their one rerun)

[test:paid] VERDICT GREEN — lane gate/pr, attempt 1
  rule 112/112 · behavior 1/1 · judge 24/24 · quarantined 0 (0 failing the lane)
  SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 0
  wall 8m17s · cost $33.17 known + 32 of 139 trial(s) cost unknown
  FLAKY 2 (failed once, passed on its one rule-16 rerun) · 2 rerun execution(s), cost $1.43

125 executed, 0 reused rule/judge records | 0 manual accepted (unscored; no score-cache credit) | $32.48 rule/judge cost | reconcile exit: 0

Profile: pr / mode dependents; selected behaviors: 113, judges: 24. Deferred to scheduled/release coverage: 143 behaviors and 0 changed prompt files. Deferred checks did not run and receive no PR-pass credit.

Derived dependents (scripts/pr-dependencies.ts):

  • scripts/lib/windows-curation.ts: 255 case(s), 24 judge(s)
Rule and judge shards
Shard Automated result Manual/unscored Executed Reused Status Cost
llm-judge/skill-llm-eval 24/24 0 24 0 ✅ $1.44
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-current 1/1 0 1 0 ✅ $1.02
e2e/skill-e2e-coverage-audit 2/2 0 2 0 ✅ $0.29
e2e-deploy/skill-e2e-deploy--setup-deploy-workflow 1/1 0 1 0 ✅ $0.13
e2e-qa-functional-fix/skill-e2e-qa-functional-fix 2/2 0 2 0 ✅ $0.99
e2e/skill-e2e-plan--plan-review-report 1/1 0 1 0 ✅ $0.57
e2e-qa-callers/skill-e2e-qa-callers--ship-exploratory-unavailable 1/1 0 1 0 ✅ $0.81
e2e-retro/skill-e2e-retro 1/1 0 1 0 ✅ $0.34
e2e/skill-e2e-shared-libs-paths--shared-libs-review-prior-coverage 1/1 0 1 0 ✅ $0.96
e2e-qa-workflow/skill-e2e-qa-workflow--qa-quick 1/1 0 1 0 ✅ $0.57
e2e-review-attribution/skill-e2e-review-attribution 3/3 0 3 0 ✅ $0.28
e2e/skill-e2e-ship-skip 1/1 0 1 0 ✅ $0.25
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-stale-before 1/1 0 1 0 ✅ $0.42
e2e/skill-e2e-shared-libs--shared-libs-review-lifecycle 1/1 0 1 0 ✅ $0.85
e2e-review-army/skill-e2e-review-army--review-army-migration-safety 1/1 0 1 0 ✅ $0.34
e2e-deploy/skill-e2e-deploy--land-and-deploy-first-run 1/1 0 1 0 ✅ $0.23
e2e/skill-e2e-shared-libs--shared-libs-read-only 1/1 0 1 0 ✅ $0.26
e2e-deploy/skill-e2e-deploy--land-and-deploy-review-gate 1/1 0 1 0 ✅ $0.23
e2e-docsync-spawned/skill-e2e-docsync-spawned 1/1 0 1 0 ✅ $0.47
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-stale-after 0/1 0 1 0 ❌ $0.41
e2e-safety-codex-consult/skill-e2e-safety-codex-consult 1/1 0 1 0 ✅ $0.29
e2e-deploy/skill-e2e-deploy--land-and-deploy-workflow 1/1 0 1 0 ✅ $0.58
e2e/skill-e2e-workflow 4/4 0 4 0 ✅ $0.99
e2e-review-army/skill-e2e-review-army--review-army-perf-n-plus-one 1/1 0 1 0 ✅ $0.23
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-late-result 1/1 0 1 0 ✅ $0.34
e2e/skill-e2e-shared-libs-paths--shared-libs-review-index-flags 1/1 0 1 0 ✅ $0.97
e2e-deploy/skill-e2e-deploy--benchmark-workflow 1/1 0 1 0 ✅ $0.22
e2e/skill-e2e-investigate-owned-completion 1/1 0 1 0 ✅ $0.07
e2e-qa-callers/skill-e2e-qa-callers--ship-exploratory-small-cli 1/1 0 1 0 ✅ $0.86
e2e-review-army/skill-e2e-review-army--review-army-json-findings 1/1 0 1 0 ✅ $0.06
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-store 1/1 0 1 0 ✅ $0.91
e2e-triage/skill-e2e-triage 1/1 0 1 0 ✅ $0.1
e2e-review-army/skill-e2e-review-army--review-army-delivery-audit 1/1 0 1 0 ✅ $0.15
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-timeout-unsettled 1/1 0 1 0 ✅ $0.18
e2e-design/skill-e2e-design--design-review-detector-shim 1/1 0 1 0 ✅ $0.13
e2e-session-intelligence/skill-e2e-session-intelligence 4/4 0 4 0 ✅ $0.37
e2e-safety-pair-agent/skill-e2e-safety-pair-agent 1/1 0 1 0 ✅ $0.1
e2e-plan-tune/skill-e2e-plan-tune 1/1 0 1 0 ✅ $0.15
e2e-qa-workflow/skill-e2e-qa-workflow--qa-bootstrap 1/1 0 1 0 ✅ $0.08
e2e-deploy/skill-e2e-deploy--canary-workflow 1/1 0 1 0 ✅ $0.1
e2e/skill-e2e-investigate-owned-termination 2/2 0 2 0 ✅ $0.17
e2e/skill-e2e-ship-hook-refresh 1/1 0 1 0 ✅ $0.1
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-launch-failure 1/1 0 1 0 ✅ $0.16
e2e-design/skill-e2e-design--plan-design-review-no-ui-scope 1/1 0 1 0 ✅ $0.07
e2e-third-party-actions/skill-e2e-third-party-actions 5/5 0 5 0 ✅ $0.44
e2e/skill-e2e-ship-hook-consent 2/2 0 2 0 ✅ $0.25
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-recovery 1/1 0 1 0 ✅ $0.39
e2e/skill-e2e-shared-libs--shared-libs-review-revalidation 1/1 0 1 0 ✅ $1.94
e2e-review/skill-e2e-review 2/2 0 2 0 ✅ $0.33
e2e-cso/skill-e2e-cso 1/1 0 1 0 ✅ $0.39
e2e-qa-functional/skill-e2e-qa-functional 2/2 0 2 0 ✅ $0.82
e2e-qa-callers/skill-e2e-qa-callers--ship-exploratory-late-input 1/1 0 1 0 ✅ $0.94
e2e-learnings/skill-e2e-learnings 1/1 0 1 0 ✅ $0.07
e2e-failure-routing/skill-e2e-failure-routing 4/4 0 4 0 ✅ $0.41
e2e-qa-workflow/skill-e2e-qa-workflow--qa-only-no-fix 1/1 0 1 0 ✅ $0.55
e2e-qa-callers/skill-e2e-qa-callers--review-exploratory-small-cli 1/1 0 1 0 ✅ $0.72
e2e-qa-callers/skill-e2e-qa-callers--ship-exploratory-plan-checks 1/1 0 1 0 ✅ $0.88
e2e/skill-e2e-test-value 3/3 0 3 0 ✅ $0.69
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-failure 1/1 0 1 0 ✅ $0.65
e2e-skillify/skill-e2e-skillify 3/3 0 3 0 ✅ $0.84
e2e/skill-e2e-shared-libs-paths--shared-libs-review-path-eligibility 1/1 0 1 0 ✅ $1.46
e2e-hermetic/skill-e2e-hermetic-canary 2/2 0 2 0 ✅ $0.02
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-completion 0/1 0 1 0 ❌ $0.96
e2e-design/skill-e2e-design--design-review-plugin-handoff 1/1 0 1 0 ✅ $0.13
e2e-review-army/skill-e2e-review-army--review-army-quality-score 1/1 0 1 0 ✅ $0.1
e2e-design/skill-e2e-design--design-review-detector-shim-dom 1/1 0 1 0 ✅ $0.22
e2e-safety-ship-evidence/skill-e2e-safety-ship-evidence 1/1 0 1 0 ✅ $0.13
e2e-browse/skill-e2e-bws 6/6 0 6 0 ✅ $0.29
e2e/skill-e2e-shared-libs--shared-libs-unsupported-git 1/1 0 1 0 ✅ $0.19
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-missing-asset 1/1 0 1 0 ✅ $0.27
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-missing-marker 1/1 0 1 0 ✅ $0.16
Fail-closed reconciliation
            27s  test/skill-e2e-design.test.ts#design-review-detector-shim
  slice 16  passed             22s  test/skill-e2e-deploy.test.ts#canary-workflow
  slice 16  passed             29s  test/skill-e2e-ship-hook-refresh.test.ts
  slice 16  passed             16s  test/skill-e2e-design.test.ts#plan-design-review-no-ui-scope
  slice 16  passed             31s  test/skill-e2e-review-army.test.ts#review-army-delivery-audit
  slice 16  passed             26s  test/skill-e2e-safety-pair-agent.test.ts
  slice 16  passed             28s  test/skill-e2e-qa-workflow.test.ts#qa-bootstrap
  slice 17  passed             11s  test/skill-e2e-learnings.test.ts
  slice 17  passed             15s  test/skill-e2e-diagram.test.ts
[test:paid] collector final outcomes: 125 executed, 0 reused; 123 passed, 2 failed, 0 manual accepted (unscored; no score-cache credit) (125 attempt records from 71 collectors; every record counts)
[test:paid] deselected testcases, zero credit (316):
[test:paid]    302  sibling case of a case shard (its own shard runs it)
[test:paid]      8  periodic-tier case (this lane runs gate)
[test:paid]      6  Bun hook placeholder of a skipped describe block (not a test)
[test:paid] VERDICT GREEN — lane gate/pr, attempt 1
  rule 112/112 · behavior 1/1 · judge 24/24 · quarantined 0 (0 failing the lane)
  SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 0
  wall 8m17s · cost $33.17 known + 32 of 139 trial(s) cost unknown
  FLAKY 2 (failed once, passed on its one rule-16 rerun) · 2 rerun execution(s), cost $1.43
[test:paid] flaky (rule 16: failed once, passed on its one rerun; green with this annotation):
  ⚑ ship-docsync-completion  rule  FLAKY: failed in attempt 1 (paid-slice-3-a1), passed on its one rerun in attempt 1 rerun (slice 1 attempt 1); rerun cost $1.02; first failure: assertion: expect(received).toEqual(expected)
  ⚑ ship-docsync-stale-after  rule  FLAKY: failed in attempt 1 (paid-slice-13-a1), passed on its one rerun in attempt 1 rerun (slice 1 attempt 1); rerun cost $0.41; first failure: assertion: expect(received).toEqual(expected)
[test:paid] exercised-path map (shadow, non-blocking): 10 MAP_GAP line(s) over 105 executed case(s) with recorded commands
  MAP_GAP document-release bin/gstack-banned-terms: non-blocking for this PR; the next census refresh widens the map (docs/TESTING_INTERNALS.md#exercised-path-map)
  MAP_GAP plan-review-report bin/gstack-plan-reality: non-blocking for this PR; the next census refresh widens the map (docs/TESTING_INTERNALS.md#exercised-path-map)
  MAP_GAP plan-review-report browse/dist/browse: non-blocking for this PR; the next census refresh widens the map (docs/TESTING_INTERNALS.md#exercised-path-map)
  MAP_GAP review-army-migration-safety bin/gstack-approval-classes: non-blocking for this PR; the next census refresh widens the map (docs/TESTING_INTERNALS.md#exercised-path-map)
  MAP_GAP ship-exploratory-late-input bin/gstack-review-plan: non-blocking for this PR; the next census refresh widens the map (docs/TESTING_INTERNALS.md#exercised-path-map)
  MAP_GAP ship-exploratory-plan-checks bin/gstack-review-plan: non-blocking for this PR; the next census refresh widens the map (docs/TESTING_INTERNALS.md#exercised-path-map)
  MAP_GAP ship-exploratory-small-cli bin/gstack-review-plan: non-blocking for this PR; the next census refresh widens the map (docs/TESTING_INTERNALS.md#exercised-path-map)
  MAP_GAP ship-exploratory-unavailable bin/gstack-review-plan: non-blocking for this PR; the next census refresh widens the map (docs/TESTING_INTERNALS.md#exercised-path-map)
  MAP_GAP test-audit-report-only design/dist/design: non-blocking for this PR; the next census refresh widens the map (docs/TESTING_INTERNALS.md#exercised-path-map)
  MAP_GAP timeline-event-flow bin/gstack-skill-end: non-blocking for this PR; the next census refresh widens the map (docs/TESTING_INTERNALS.md#exercised-path-map)
[test:paid] report: every planned shard accounted and passed

Sliced lane: planner → duration-packed executors → fail-closed report. Behavior cases run a pre-registered 3-trial panel (PASS at 2/3 with no contract violation); a PASS 2/3 is shown with its failed trial, never as a clean pass. Reused results retain their original provenance and expiry.

Flaky: failed once, passed on its one rerun (rule 16)

⚑ ship-docsync-completion  rule  FLAKY: failed in attempt 1 (paid-slice-3-a1), passed on its one rerun in attempt 1 rerun (slice 1 attempt 1); rerun cost $1.02; first failure: assertion: expect(received).toEqual(expected)
⚑ ship-docsync-stale-after  rule  FLAKY: failed in attempt 1 (paid-slice-13-a1), passed on its one rerun in attempt 1 rerun (slice 1 attempt 1); rerun cost $0.41; first failure: assertion: expect(received).toEqual(expected)

Each failed and passed on its rerun in run 40749421675 at the attempts named above. Logged for the daily flaky-fix pass in this run's flake-records-a1 artifact (docs/TESTING_INTERNALS.md#flaky-reruns).

Dispatched narrow screen (gstack-ship-queue ready --screen) of merge commit e5ac31871f09fa2df84a15d0cb7c8ed661eb4a99.

garrytan and others added 3 commits October 28, 2026 18:27
…merge base (screen 40737741454 ship-exploratory-unavailable)

The review plan dispatches the state-machines specialist, and review-army tells the
parent to write the merge-base sha into its diff command. Both screen trials failed
on that child's git diff <merge-base sha>. Also: cite the deadline loop beside the
identity_* cookie-auth cases (GStack Minder accepted the 400 ms narrowing).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@github-actions github-actions Bot changed the title fix: repair three flaky gate cases from the 2026-10-28 flaky pass (index flags, exploratory QA, cookie auth) v1.91.302.0 fix: repair three flaky gate cases from the 2026-10-28 flaky pass (index flags, exploratory QA, cookie auth) Oct 29, 2026
@capy-ai
capy-ai Bot merged commit 8b4f5aa into main Oct 29, 2026
63 checks passed
capy-ai Bot added a commit that referenced this pull request Nov 1, 2026
…c detector gaps (2026-10-29 flaky pass, GSTA-162) (#3606)

* fix(flaky): qa-callers parents hand children literal commands and paths; Step 9 gate scan joins diff discovery

Nine paid ship qa-callers reds after #3553 (late-input, plan-checks, small-cli) had two causes,
replayed from their retained captures (2026-10-29 flaky pass, GSTA-162):
- six trials: the parent paraphrased its specialist prompt and the child ran the one refused call
  (git -C <dir> diff, or a cd into the checklist directory);
- three late-input trials hit error_max_turns at 25/25 after running Step 9's gate scan only after
  the receipt, or never.

The session prompt now tells the parent the child starts in the product directory, to write each
command exactly as it must run and each Read as one literal path; the ship scheduling contract lists
Step 9's review-plan command and gate-scan block with the first diff discovery. The detector is
unchanged. Seven new replay tests fail with the fixture prompt stashed.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(flaky): docsync detectors accept a read given --help, the cap gate's wrong-path 127 and a full read before a failed separator; only the accepted docs attempt is parsed strictly

Replayed from the 2026-10-29 window's ship-docsync reds (GSTA-162):
- current 40822033640, completion 40815828320, store 40809051510: the parent joined
  `cat <candidate> --help 2>&1 | head -N` onto a line of reads; GNU cat printed its usage and
  read nothing. A declared read command given --help is neutral; its operands are still checked.
- stale-after 40749421675: the cap gate typed at the shard root exited 127, then ran at the
  declared path. The wrong-path 127 excuse now covers the cap gate as well as the candidate.
- failure 40828580338: phase.md printed in full, then zsh failed the `echo ======` separator.
  The read counts when the line's first unpiped cat printed the whole file first.
- completion 40805543956: the parent rejected attempt 1 (no documentation_section) and re-audited;
  the case now parses only the accepted attempt strictly.
Four public fixtures, registered as free fixtures.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* CHANGELOG: [Unreleased] entry for the 2026-10-29 flaky pass

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: re-pin the ship-docsync completion/current detector-inventory row to its moved lines

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant