Skip to content

v1.92.33.0 fix(flaky): 2026-10-31 flaky pass — paid detector gaps, Windows cold first launches, fixture races, ubicloud pack index copy (GSTA-199) - #3697

Merged
capy-ai[bot] merged 10 commits into
mainfrom
capy/gsta199-flaky
Nov 1, 2026
Merged

capy-ai[bot] merged 10 commits into
mainfrom
capy/gsta199-flaky

Conversation

@garrytan

@garrytan garrytan commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner
{
  "schema_version": 1,
  "pr": 3697,
  "head": "ed4830a44a51167586431c88367660f628c2324d",
  "version": "1.91.369.0",
  "base": "main",
  "gate": {
    "typecheck": {
      "pass": 1,
      "fail": 0,
      "skip": 0
    },
    "tests": {
      "pass": 10614,
      "fail": 0,
      "skip": 0
    }
  },
  "session_kind": "unattended",
  "head_tree": "002ce7c57ac50fd8afdee886e0d827708c1de772",
  "product_patch_id": "675bf8f0a7f594de6e483499418ea8f339a4add5",
  "gstack_version": "1.91.366.0",
  "review_depth": {
    "depth": "light",
    "class": "code",
    "runtime_lines": 7,
    "rules": "150435b0a543019a",
    "base": "44eeeb3c9b8011f17a44cb234fdf052afef660f8",
    "registry": "44eeeb3c9b8011f17a44cb234fdf052afef660f8",
    "gate_scan": {
      "tagged": 0,
      "head": "ed4830a44a51167586431c88367660f628c2324d"
    },
    "reasons": [
      "runtime=7<=50"
    ]
  },
  "pregate": "requires-remote cleared @002ce7c57ac5 (7 checks, current for ed4830a44a51)",
  "requires_remote": [
    "free-tests.yml/macos-named-regressions: https://github.com/garrytan/gstack/actions/runs/41215919704",
    "windows-free-tests.yml/windows-free-shard: https://github.com/garrytan/gstack/actions/runs/41215919638",
    "pregate.json/remote_gate: remote-gate tree=002ce7c57ac5 tests=10614 receipt=/home/user/.capy/work/gsta199/ship2/remote-gate-receipt.log sha256=8ba50726774f"
  ],
  "issues": {
    "fixes": [],
    "refs": [],
    "flakes": []
  },
  "docs": {
    "status": "current",
    "scope": "AGENTS.md, CLAUDE.md, CONTRIBUTING.md, README.md and docs/ (133 docs) against base 02da1a05 to head ed4830a"
  },
  "review_loop": {
    "passes": 1,
    "blocking_found": 0,
    "deferred": 0,
    "noted": 0,
    "fixed": 0,
    "scope": "frozen",
    "est_input_tokens": 26945,
    "dispatches": {
      "specialist-batched": 1
    }
  }
}

What changed

Two paid-eval detectors that failed correct runs now credit them: the coverage-audit two-file grep display and the plan-design lettered 'A) All 7' option

Two paid-eval detectors that failed correct runs now credit them. Before is the base detectors replaying this PR's saved captures; after is this branch.

Before/after as text: Two paid-eval detectors that failed correct runs now credit them: the coverage-audit two-file grep display and the plan-design lettered 'A) All 7' option
--- Before · before-shot.txt · captured output
+++ After · after-shot.txt · captured output
@@ ⋯ 1 unchanged ⋯ @@
 (pass) coverage audit native evidence: captured owned-cd chains  > a chain that never reads an owned marker file gets no credit for it
 (pass) coverage audit native evidence: captured owned-cd chains  > wc -l counts only literal in-cwd files and stays inside its line bounds
 (pass) coverage audit native evidence: a separator that also appears in neighboring output  > the captured echo ---- read and its echo --- form both credit the owned files
+(pass) coverage audit native evidence: an escaped grep over two literal files after the reads  > both complete reads before a two-file escaped grep display are credited and the recorded run passes
+(pass) UI gate seeds the exact target and accepts Design progress with an unselected no-UI alternative (captured)
+(pass) UI gate seeds the exact target and accepts Design progress with an unselected no-UI alternative (captured-rated)
+(pass) UI gate seeds the exact target and accepts Design progress with an unselected no-UI alternative (captured-run)
+(pass) UI gate rejects captured-no-focus and removes its fixture
+(pass) UI gate rejects captured-focus-after-finding and removes its fixture
+(pass) UI gate rejects captured-rated-no-focus and removes its fixture
+(pass) UI gate rejects captured-run-no-focus and removes its fixture
-(fail) coverage audit native evidence: an escaped grep over two literal files after the reads  > both complete reads before a two-file escaped grep display are credited and the recorded run passes
-(fail) UI gate seeds the exact target and accepts Design progress with an unselected no-UI alternative (captured)
-(fail) UI gate seeds the exact target and accepts Design progress with an unselected no-UI alternative (captured-rated)
-(fail) UI gate seeds the exact target and accepts Design progress with an unselected no-UI alternative (captured-run)
 

Why

The daily flaky pass for 2026-10-30 13:00Z to 2026-10-31 13:00Z found 9 paid gate flakes and 30 free-test retry passes across 507 completed CI runs. Each one cost a rerun and a slower train. Eleven of them now pass on correct runs, and six more record enough to name their cause the next time they fail.

Blast radius

Every change is in tests, test helpers and fixtures, except one line in scripts/ubicloud/ubi-runner.sh (cmd_pack copies .git/index with cat instead of cp). That copy only reads the index, so its output is the same bytes; a loop that swaps the index during 300 packs failed 10 times before the change and 0 after (script ran). The detector edits only widen what counts as a correct reply, and each one keeps negative controls that still count zero (replay tests ran).

Approvals

No approvals recorded for branch capy/gsta199-flaky.

Summary

The nightly pass sorted every flake by cause. Three paid detectors missed correct replies. They did not accept a lettered "A) All 7" option, a grep display over two files, or a "not installed but can be downloaded" sentence. Each now credits the reply, with the captured run saved as a fixture and the wrong forms still rejected. On Windows, most free-test reds came from a binary's slow first launch on a fresh runner, so four test files launch jq, awk or PowerShell once before their timed calls; every kill time is unchanged. Two races are gone. The publication-lock test no longer reads a half-written ready file, and an EBUSY cleanup now waits out the runner's own scanner. Seven tests and helpers now log what the next failure needs, including the judge's response bytes and the time of its last byte. Thresholds, trial counts and case kinds are unchanged, and no retry was added.

Test Coverage

Every repaired detector has a replay test built from the run that failed, and each one fails without its fix: plan-design-with-ui (36/36 with the fix), coverage-audit (231/231), third-party-actions (60/60, 59 pass and 1 fail without). The judge byte counter has a new test in test/llm-judge-stream.test.ts (21/21). The ubi-runner change is covered by 11 existing tests that reference the script.

Tests ran: typecheck count unavailable
Tests ran: free tier on the remote pregate 10614 (0 failed, tree 002ce7c57ac5)

Pre-Landing Review

Pre-Landing Review: No issues found. This is the review of the final head, ed4830a. The batched specialist (testing, security, state-machines, red-team), the core checklist, the native adversarial pass and the outside voice all returned no findings. An earlier /ship run on ec8b478 fixed 1 test gap from pass 1 and 4 findings from Step 11: the passive-pitch clause scan went from quadratic to linear, grep operands are now tokenized, and the ship-budget harness diagnostics were fixed.
REVIEW_LOOP: passes=1 blocking_found=0 deferred=0 noted=0 fixed=0 scope=frozen dispatches=1 est_input_tokens=26945
Gate edits: none detected (45 listed, 21 read; patterns: RH-1,RH-3,RH-4,RH-12,RH-13,RH-14,RH-15,RH-16; unlisted idioms: none)
Outside review: openai/gpt-6.1-sol, clean (OUTSIDE_STATUS: completed).

Review notes

  • scripts/ubicloud/ubi-runner.sh cmd_pack fetches refs before it copies the index, so a commit that lands between the two could leave staged blobs out of the pack. This predates the PR and sits outside the diff.

Exploratory QA

PASS with no defects at ed4830a. The ubicloud runner pack probes ran against local repos only, with no VM. The other test changes are covered by their replay tests.

Design Review

No frontend files changed — design review skipped.

Eval Results

Focused paid reruns, one trial each, as each case's kind requires: plan-design-with-ui-scope PASS (about $0.80, cost not printed for PTY runs), review-coverage-audit PASS ($0.13), review/SKILL.md workflow judge PASS ($0.06, 3 samples in 100 s through the wrapped fetch), tpa-absent-darwin PASS ($0.08), tpa-absent-linux PASS ($0.09). About $1.16 total.

Greptile Review

No Greptile comments.

Plan Completion

Plan completion audit: not run (fix-lane exemption: an already-diagnosed flaky list; the dispositions are in the crew flaky ledger).

Verification Results

Every CI check passes at ed4830a. This PR used to also fix a lost SIGCONT in test/cso-snapshot-state.test.ts. That fix is now out, because the file's path starts the Stage CSO Runtime Images workflow. That workflow fails on an upstream tag move (SOURCE_TAG_MOVED: docker.io/docker/scout-sbom-indexer:1, run 41212876036) until the SBOM generator is re-pinned (GST-495). The fix comes back in a later flaky pass.

TODOS

No TODO items completed in this PR.

Documentation

Status: current. The selected release changes already have accurate docs, and no edits were needed.

Scope: base 02da1a05 to head ed4830a on capy/gsta199-flaky: 29 committed files (test fixes and diagnostics, scripts/ubicloud/ubi-runner.sh, scripts/harness-version.json, one changelog fragment and one docs paragraph). The candidate had no staged, unstaged or new files. Every tracked Markdown or template doc that names a changed surface was checked against the diff.

Documentation health:

  • docs/TESTING_INTERNALS.md: Current. The llm-judge-aborted field list (bytesReceived, lastByteMs) matches test/helpers/llm-judge.ts.
  • changelog.d/capy-gsta199-flaky.md: Current (protected, not edited). No leftovers from the reverted cso-snapshot change. Open question for the author: the headline says "six more" tests now name their cause, but the Changed section lists seven tests and helpers.
  • docs/evals/detector-inventory.md: Current as a dated snapshot pinned to HEAD cbcccf9. Its line references trail the changed test helpers by a few lines.
  • README.md, CONTRIBUTING.md, AGENTS.md, CLAUDE.md: Current. No user-facing commands, skills or setup steps changed.

Documentation debt:

  • Common gap: warmFirstExec (test/helpers/warm-first-exec.ts) is described in the helper's own comment and in the changelog fragment, but the Windows free-lane docs have no how-to for contributors writing timed Windows tests. /document-generate could add one.

Diagram drift: none.

Test plan

  • bun run typecheck: exit 0
  • Remote pregate (gstack-pregate --stage tests --remote, Ubicloud): 10614 tests, 0 failed, tree 002ce7c57ac5
  • Replay tests for each repaired detector: fail without the fix, pass with it
  • Focused paid reruns: 5 of 5 PASS

🤖 Generated with Claude Code

garrytan and others added 8 commits October 31, 2026 14:05
…irst launches, cleanup and settlement races, ubicloud pack index copy (GSTA-199)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…check reads the spawn timeout unchanged

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…scan, tokenized grep operands, harness diagnostic reset and early stderr

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…udge byte counter

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

garrytan and others added 2 commits October 31, 2026 15:30
…generator re-pin (GST-495)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@github-actions github-actions Bot changed the title fix(flaky): 2026-10-31 flaky pass — paid detector gaps, Windows cold first launches, fixture races, ubicloud pack index copy (GSTA-199) v1.92.33.0 fix(flaky): 2026-10-31 flaky pass — paid detector gaps, Windows cold first launches, fixture races, ubicloud pack index copy (GSTA-199) Nov 1, 2026
@capy-ai
capy-ai Bot merged commit 1cac25c into main Nov 1, 2026
62 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant