Repository navigation
v1.92.33.0 fix(flaky): 2026-10-31 flaky pass — paid detector gaps, Windows cold first launches, fixture races, ubicloud pack index copy (GSTA-199) - #3697
Merged
Conversation
…irst launches, cleanup and settlement races, ubicloud pack index copy (GSTA-199) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…check reads the spawn timeout unchanged Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…scan, tokenized grep operands, harness diagnostic reset and early stderr Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…udge byte counter Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
|
😎 This pull request was merged. |
…generator re-pin (GST-495) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
4 tasks done
4 tasks done
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Two paid-eval detectors that failed correct runs now credit them. Before is the base detectors replaying this PR's saved captures; after is this branch.
Before/after as text: Two paid-eval detectors that failed correct runs now credit them: the coverage-audit two-file grep display and the plan-design lettered 'A) All 7' option
Why
The daily flaky pass for 2026-10-30 13:00Z to 2026-10-31 13:00Z found 9 paid gate flakes and 30 free-test retry passes across 507 completed CI runs. Each one cost a rerun and a slower train. Eleven of them now pass on correct runs, and six more record enough to name their cause the next time they fail.
Blast radius
Every change is in tests, test helpers and fixtures, except one line in
scripts/ubicloud/ubi-runner.sh(cmd_packcopies.git/indexwithcatinstead ofcp). That copy only reads the index, so its output is the same bytes; a loop that swaps the index during 300 packs failed 10 times before the change and 0 after (script ran). The detector edits only widen what counts as a correct reply, and each one keeps negative controls that still count zero (replay tests ran).Approvals
No approvals recorded for branch capy/gsta199-flaky.
Summary
The nightly pass sorted every flake by cause. Three paid detectors missed correct replies. They did not accept a lettered "A) All 7" option, a grep display over two files, or a "not installed but can be downloaded" sentence. Each now credits the reply, with the captured run saved as a fixture and the wrong forms still rejected. On Windows, most free-test reds came from a binary's slow first launch on a fresh runner, so four test files launch jq, awk or PowerShell once before their timed calls; every kill time is unchanged. Two races are gone. The publication-lock test no longer reads a half-written ready file, and an EBUSY cleanup now waits out the runner's own scanner. Seven tests and helpers now log what the next failure needs, including the judge's response bytes and the time of its last byte. Thresholds, trial counts and case kinds are unchanged, and no retry was added.
Test Coverage
Every repaired detector has a replay test built from the run that failed, and each one fails without its fix: plan-design-with-ui (36/36 with the fix), coverage-audit (231/231), third-party-actions (60/60, 59 pass and 1 fail without). The judge byte counter has a new test in
test/llm-judge-stream.test.ts(21/21). The ubi-runner change is covered by 11 existing tests that reference the script.Tests ran: typecheck count unavailable
Tests ran: free tier on the remote pregate 10614 (0 failed, tree 002ce7c57ac5)
Pre-Landing Review
Pre-Landing Review: No issues found. This is the review of the final head, ed4830a. The batched specialist (testing, security, state-machines, red-team), the core checklist, the native adversarial pass and the outside voice all returned no findings. An earlier /ship run on ec8b478 fixed 1 test gap from pass 1 and 4 findings from Step 11: the passive-pitch clause scan went from quadratic to linear, grep operands are now tokenized, and the ship-budget harness diagnostics were fixed.
REVIEW_LOOP: passes=1 blocking_found=0 deferred=0 noted=0 fixed=0 scope=frozen dispatches=1 est_input_tokens=26945
Gate edits: none detected (45 listed, 21 read; patterns: RH-1,RH-3,RH-4,RH-12,RH-13,RH-14,RH-15,RH-16; unlisted idioms: none)
Outside review: openai/gpt-6.1-sol, clean (OUTSIDE_STATUS: completed).
Review notes
scripts/ubicloud/ubi-runner.shcmd_packfetches refs before it copies the index, so a commit that lands between the two could leave staged blobs out of the pack. This predates the PR and sits outside the diff.Exploratory QA
PASS with no defects at ed4830a. The ubicloud runner
packprobes ran against local repos only, with no VM. The other test changes are covered by their replay tests.Design Review
No frontend files changed — design review skipped.
Eval Results
Focused paid reruns, one trial each, as each case's kind requires: plan-design-with-ui-scope PASS (about $0.80, cost not printed for PTY runs), review-coverage-audit PASS ($0.13), review/SKILL.md workflow judge PASS ($0.06, 3 samples in 100 s through the wrapped fetch), tpa-absent-darwin PASS ($0.08), tpa-absent-linux PASS ($0.09). About $1.16 total.
Greptile Review
No Greptile comments.
Plan Completion
Plan completion audit: not run (fix-lane exemption: an already-diagnosed flaky list; the dispositions are in the crew flaky ledger).
Verification Results
Every CI check passes at ed4830a. This PR used to also fix a lost SIGCONT in
test/cso-snapshot-state.test.ts. That fix is now out, because the file's path starts the Stage CSO Runtime Images workflow. That workflow fails on an upstream tag move (SOURCE_TAG_MOVED: docker.io/docker/scout-sbom-indexer:1, run 41212876036) until the SBOM generator is re-pinned (GST-495). The fix comes back in a later flaky pass.TODOS
No TODO items completed in this PR.
Documentation
Status: current. The selected release changes already have accurate docs, and no edits were needed.
Scope: base
02da1a05to headed4830aoncapy/gsta199-flaky: 29 committed files (test fixes and diagnostics,scripts/ubicloud/ubi-runner.sh,scripts/harness-version.json, one changelog fragment and one docs paragraph). The candidate had no staged, unstaged or new files. Every tracked Markdown or template doc that names a changed surface was checked against the diff.Documentation health:
docs/TESTING_INTERNALS.md: Current. Thellm-judge-abortedfield list (bytesReceived,lastByteMs) matchestest/helpers/llm-judge.ts.changelog.d/capy-gsta199-flaky.md: Current (protected, not edited). No leftovers from the reverted cso-snapshot change. Open question for the author: the headline says "six more" tests now name their cause, but the Changed section lists seven tests and helpers.docs/evals/detector-inventory.md: Current as a dated snapshot pinned to HEADcbcccf9. Its line references trail the changed test helpers by a few lines.README.md,CONTRIBUTING.md,AGENTS.md,CLAUDE.md: Current. No user-facing commands, skills or setup steps changed.Documentation debt:
warmFirstExec(test/helpers/warm-first-exec.ts) is described in the helper's own comment and in the changelog fragment, but the Windows free-lane docs have no how-to for contributors writing timed Windows tests./document-generatecould add one.Diagram drift: none.
Test plan
bun run typecheck: exit 0gstack-pregate --stage tests --remote, Ubicloud): 10614 tests, 0 failed, tree 002ce7c57ac5🤖 Generated with Claude Code