Skip to content

Free-suite CI survives a slow Ubuntu mirror, and four flaky tests and eval checks stop failing (GST-3206) - #4530

Merged
capy-ai[bot] merged 5 commits into
mainfrom
flaky-fix-2026-11-19
Nov 19, 2026
Merged

capy-ai[bot] merged 5 commits into
mainfrom
flaky-fix-2026-11-19

Conversation

@garrytan

@garrytan garrytan commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Free-suite CI stops failing before any test runs when archive.ubuntu.com is slow, and four tests that failed and then passed in the last day are fixed at their cause.
Why: 27 setup failures and 5 flaky reds on 2026-11-19 cost reruns and blocked READY checks across PRs and main.
Nothing to do: the first run on each runner image fills the new package cache.

What changed
  • CI package cache (.github/workflows/free-tests.yml, free-suite and quick-bun-default). Playwright's --with-deps downloads about 38 MB of system packages every run. On 2026-11-19 the mirror served some runners at 85 to 280 kB/s: 17 of 456 first install attempts hit the 5-minute limit, and the retry then died in about 6 s on the dpkg lock that the killed attempt's apt-get still held (run 44485949701, job 135438838194). 5 more jobs failed in the Xvfb/gate-tools apt step on fetch errors. Now apt waits up to 240 s for its lock and keeps what it downloads; those .deb files are cached under a key built from a hash of the runner's installed package list and the Playwright version. A later run with the same key installs them with dpkg -i, runs playwright install chromium without --with-deps, and the Xvfb step finds everything installed. A failed offline install falls back to today's path. Step timeouts and the single retry are unchanged.
  • first-use (lib/first-use.ts). Several user drives were listed in readdirSync order in the refusal text; now sorted. Failed on main and on /sync-gbrain's read check works when the first page is over 64 KB (#4396) #4497 and On Capy, /ship no longer edits or comments on pull requests as you (GST-3183) #4517.
  • gstack-lease test (test/gstack-lease.test.ts). The test read the lock's owner file as soon as it existed, but gstack_lock_take creates it with O_EXCL and writes the line after, so it could read "" (main run 44512215095). The test now waits for a complete line, and its shell hook holds the write open for 0.3 s so the window is hit every run.
  • Read-only command grammar (test/helpers/read-only-commands.ts). A backslash-quote inside double quotes is a literal quote in bash and zsh; the tokenizer refused it and failed ship-docsync-failure on census 44321344888. Only that escape is admitted; \$, backtick and \\ stay refused.
  • Coverage audit (test/helpers/coverage-audit-evidence.ts). Claude Code keeps a Bash call's final cwd. The checker now tracks a proved session cwd, so cd <cwd>/review; ls ... then cd .. && cat -n src/billing.ts && ... counts (review-coverage-audit, census 44351657692). The cwd becomes unknown after a failed call, any unrecognized command shape, a subagent's Bash call or overlapping Bash calls; the absolute-cwd rule is unchanged.
How it was tested
  • New first-use test spies readdirSync into reverse order: fails on main with the CI symptom, passes with the sort. 23/23.
  • gstack-lease: the hardened test fails on the old wait with Received: "" (the CI symptom) and passes with the new wait; whole file 40/40.
  • Workflow wiring test updated for the new steps and extended with the cache contract; 25/25 with the sandbox test. actionlint 1.7.11 clean.
  • Paid checker repairs: free replays of the captured census events are red on main with the recorded symptom and green after; 6 new regressions fail without the helper changes, with 40 negative controls that stay red. 204 + 202 + 148 pass. No paid runs.
  • Typecheck clean; test typecheck ratchet: none new.
Agent notes

Daily flaky-fix pass GSTA-654 (GST-3206). Ledger: handoffs/flaky-ledger.md in the GStack Minder drive. ship-docsync-current (census 44432023197) stays Open: the child omitted documentation_section under clear instructions and the fixture does not declare the repair-prompt / result --repair-of commands documentation.md asks for, so the parent stopped as blocked; that repair needs a paid proof.

Plan: exempt: flaky

Review (one pass on fa1bc25, tree 7681981, no fix round): plan class=gate, 5 readers (specialist batch, red team, adversarial, second-model review on gpt-6.1-sol, targeted gate reader). Pre-Landing Review: 2 issues — 0 auto-fixed, 2 asked (0 fixed, 2 skipped), 0 deferred, 0 noted. Gate edits: 23 listed, 11 read, 0 findings — 0 kept, 0 restored, 0 open (the two RH-14 hunks are the new continue-on-error offline install steps; the next step reads their outcome).

Open notes (none blocks):

  1. A partly failed offline dpkg -i can leave packages unconfigured, so the --with-deps fallback's apt-get may refuse (specialist, adversarial).
  2. A cache entry whose debs fail to install is never re-saved, so that image silently takes the network path (red team).
  3. The offline step has no timeout-minutes and no DEBIAN_FRONTEND=noninteractive (adversarial).
  4. The 240 s apt lock wait uses most of the retry's 5 minutes (adversarial).
  5. Resolved: "27 jobs failed" was recounted from raw job data (22 in the Playwright retry step, 5 in the Xvfb step, across 5 runs); "17 of 456" is first-attempt timeouts in a 40-run sample. Both stand.

Exploratory QA: pass (cache_key output in both jobs; --with-deps fallback for every apt_offline outcome other than success).

Tests: targeted lane 646 pass, 1 skip, 0 fail; full free suite carried from CI run 44549845101 at tree 7d7baac2115c.

Documentation:
Scope: the diff from 1730386 to 558673e, which changes the free-tests workflow (apt/dpkg caching for Playwright packages), a sorted drive listing in lib/first-use.ts, and several test helpers and fixtures. Result: current. The changelog fragment already covers every user-visible effect, and no README, guide or reference doc describes the changed behavior. Debt: I grepped Markdown for free-tests.yml and listed nine referencing files, but read only the changelog fragment in full. I did not read CONTRIBUTING.md or docs/TESTING_INTERNALS.md in full, so a stale description of the workflow's apt steps there is unchecked. I audited no other docs.

{
  "schema_version": 1,
  "pr": 4530,
  "head": "558673ebed4a79ef4b0d349316abb3bb43620738",
  "version": "1.101.0.0",
  "base": "main",
  "gate": {
    "targeted": {
      "pass": 646,
      "fail": 0,
      "skip": 1
    },
    "test": {
      "pass": 38311,
      "fail": 0,
      "skip": 0
    }
  },
  "ci_carried": [
    {
      "lane": "test",
      "run": 44549845101,
      "attempt": 1,
      "tree": "7d7baac2115c630639a3534b9c4f23f7e54e65d3"
    }
  ],
  "calls": 24,
  "session_kind": "unattended",
  "run": "ship-4530-fa1bc25",
  "head_tree": "7d7baac2115c630639a3534b9c4f23f7e54e65d3",
  "product_patch_id": "f184df0bedb42d779f87f2bb09f0d565279c4ceb",
  "gstack_version": "1.101.0.0",
  "review_depth": {
    "depth": "full",
    "class": "gate",
    "runtime_lines": 104,
    "rules": "1afa38a4be3b5eef",
    "base": "17303862d0cea498f7073d8bc68c3932de273270",
    "registry": "17303862d0cea498f7073d8bc68c3932de273270",
    "gate_scan": {
      "tagged": 2,
      "head": "558673ebed4a79ef4b0d349316abb3bb43620738"
    },
    "reasons": [
      "gate-class",
      "runtime=104"
    ],
    "risk": [],
    "mode": "panel"
  },
  "pregate": "requires-remote cleared @7d7baac2115c (8 checks, current for 558673ebed4a)",
  "requires_remote": [
    "free-tests.yml/cso-macos-launcher: https://github.com/garrytan/gstack/actions/runs/44549845101/attempts/1",
    "free-tests.yml/macos-named-regressions: https://github.com/garrytan/gstack/actions/runs/44549845101/attempts/1",
    "free-tests.yml/cso-windows-launcher: https://github.com/garrytan/gstack/actions/runs/44549845101/attempts/1",
    "pregate.json/remote_gate: remote-gate tree=7d7baac2115c tests=38311 skipped=67 runs=44549845101/1 profiles=ci-reuse/free-tests heads=558673ebed4a verified=2026-11-19T15:10:01.057Z"
  ],
  "issues": {
    "fixes": [],
    "refs": [],
    "flakes": []
  },
  "docs": {
    "status": "current",
    "scope": "docs/, README.md, AGENTS.md, CLAUDE.md, CONTRIBUTING.md against the 13-file diff (CI apt cache, first-use sort, eval-checker test helpers); changelog fragment read in full, referencing docs listed by grep"
  },
  "review_loop": {
    "passes": 1,
    "blocking_found": 0,
    "deferred": 0,
    "noted": 4,
    "fixed": 0,
    "scope": "frozen",
    "rounds": 1,
    "dispatches": {
      "specialist": 1,
      "red-team": 1
    }
  },
  "gate_dispositions": [],
  "review_targeted": {
    "classes": [
      "gate"
    ],
    "rules": "23eb808423669ba0",
    "brief_sha": "7f92a0d0a8319c3c0e076fb3b1a4689c45120b2be2c14dfc7d4cb03aafa9d6a5",
    "product_patch_id": "f184df0bedb42d779f87f2bb09f0d565279c4ceb",
    "status": "done",
    "findings": 0,
    "raised_by": "targeted"
  }
}

Open in Capy

garrytan and others added 2 commits November 19, 2026 14:21
…sts stop racing

- CI caches the system packages Playwright and the gate tools download, keyed on the runner's installed package list and the Playwright version, and apt waits for a killed attempt's dpkg lock instead of failing the retry.
- first-use names several user drives in sorted order on every filesystem.
- gstack-lease's lock test waits for the owner line, not just the owner file.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…runs

- The read-only command grammar reads a backslash-quote inside double quotes as a literal quote, as bash and zsh do; other escapes there stay refused (ship-docsync-failure, census 44321344888).
- The coverage audit follows a proved session cwd, so cd into a subfolder and then cd .. && cat in the next call counts as reading the files (review-coverage-audit, census 44351657692).
- changelog fragment.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

garrytan and others added 2 commits November 19, 2026 14:38
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…reads it as a plain output

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@capy-ai
capy-ai Bot merged commit 3cccf07 into main Nov 19, 2026
53 checks passed
@garrytan garrytan mentioned this pull request Nov 19, 2026
capy-ai Bot added a commit that referenced this pull request Nov 19, 2026
- `gstack-artifact validate` no longer passes a run.json that gstack-autoplan never wrote. (#4494)
- /ship's changed-number check no longer stops on tests that only share a word with the change (#4478). (#4496)
- On Capy, /ship no longer edits or comments on pull requests as you. (#4518)
- The destructive-command check stops asking about scratch deletes and catches the real damage it used to miss. (#4513)
- The destructive-command check reads commands more carefully, closing gaps found after the last release. (#4525)
- The Codex readiness check and the second-model review now pick the same model for a custom Codex provider (GST-3089). (#4491)
- gstack's Codex hooks work when your Codex folder's path has a `(`, `&` or other shell character in it. (#4528)
- Uninstalling gstack keeps your own Codex hooks trusted. (#4528)
- Codex hook setup matches Codex more closely. (#4528)
- On Windows, setup copies the Claude Code hook code that gstack's Codex hooks import (#4528)
- /review's checklist on Codex reads shell commands more carefully. (#4528)
- On Codex, gstack now suggests the right skill for a prompt and stops destructive commands before they run. (#4512)
- On Codex, /review is now held once when it tries to finish without its adversarial pass or its report, as it already is on Claude Code. (#4522)
- On Codex, gstack's skill suggestion now names the skill the way Codex lists it. (#4523)
- On Codex, gstack's suggested prompts and docs now name skills the way Codex lists them. (#4524)
- /review on Codex with gpt-6.1-sol is no longer held back for a second-model pass it already ran. (#4526)
- On Codex, office-hours, plan-design-review and gstack-upgrade work again when you name them. (#4510)
- gstack knows when Codex is running unattended. (#4510)
- Codex reviewers start with a clean context. (#4510)
- Docs no longer say Codex has no hook support. (#4510)
- A command whose output was cut off no longer counts as a finished read. (#4492)
- A folder marked as published data can no longer hide source code or agent files from /ship's review. (#4516)
- Correction to the on-by-default helper hooks entry: the hook still routes only to /review, /qa and /verify, so skills you've installed yourself aren't suggested yet. With gstack's skills it routes 14 of 60 real bug reports (was 17), and long prompts are no longer skipped. (#4505)
- /sync-gbrain's read check now works when the first page in your code source is over 64 KB. (#4497)
- Free-suite CI no longer fails before any test runs when Ubuntu's package mirror is slow. (#4530)
- Syncing a second repo no longer makes the first one ask for /sync-gbrain again. (#4532)
- gstack's helper hooks are now on by default in Claude Code, OpenClaw and Hermes, and the prompt-routing hook fires on real bug reports. (#4503)
- `gstack-pregate clear --receipt` accepts a remote gate run from before a merged release when the release only regenerated a declared release output. (#4483)
- The second-model review now keeps the reviewer's own result, and reports a used-up budget as a budget stop. (#4515)
- When the reviewer wrote its review to the result file it was asked for, Codex then replaced that file with its short closing message ("Done, I wrote the review"), so the review was thrown away and reported as missing its markers. (#4515)
- When the provider refuses the call with HTTP 402 Payment Required (usually a metering proxy that hit its allowance, or an account out of credits or quota), gstack used to record only "exit 1". (#4515)
- The pre-commit /verify check now fails a check that times out, and its checks share one time budget (GST-3201). (#4527)
- /ship's review collector no longer reads a reviewer's own scope list or list of passed checks as findings (GST-3201). (#4527)
- Merge-queue stack tests fail when git cannot list a PR's added test files (GST-3201). (#4527)
- /ship can now review results PRs that publish raw session data. (#4511)
- /ship's review pass works when the checkout you ship isn't the project's default checkout. (#4507)
- A second review round no longer reports done when nothing was reviewed. (#4520)
- The prompt-routing hook now suggests skills you install yourself, and no longer sends ordinary bug reports to /qa (GST-3123). (#4506)
- gstack's script runner and merge-queue checks now also refuse cut-off output. (#4502)
- A failed second-model review reports the cause that actually stopped it, and the runner never deletes its own prompt. (#4519)
- `gstack-ship-queue ready` now fills in the trial merges and the second-model trail read instead of printing lines to fill. (#4529)
- Five small ship-tool fixes. (#4529)
- A ruling the owner gives while a `/ship` run is open can be relayed. (#4529)
- /ship no longer kills a working test run as stalled when the test command starts its workers in their own session. (#4533)
- /ship no longer stops a review because the state-machine reader started its reply the way its instructions say. (#4504)
- When Claude Code runs /verify right before a commit, it now gets a quick check instead of the full fix loop (GST-3150). (#4508)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant