Skip to content

v1.91.255.0 fix(ship-queue): fill trains at 3 READY PRs or one hour, and price every case shard's trial - #3478

Merged
capy-ai[bot] merged 8 commits into
mainfrom
train-fill-and-cost
Oct 26, 2026
Merged

capy-ai[bot] merged 8 commits into
mainfrom
train-fill-and-cost

Conversation

@garrytan

@garrytan garrytan commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner
{
  "schema_version": 1,
  "pr": 3478,
  "head": "2724f30ee090659c2b719415c81320a0341259b3",
  "version": "1.91.248.0",
  "base": "main",
  "gate": {
    "focused": {
      "pass": 341,
      "fail": 0,
      "skip": 0
    },
    "guards": {
      "pass": 75,
      "fail": 0,
      "skip": 0
    },
    "windows-curation": {
      "pass": 122,
      "fail": 0,
      "skip": 0
    },
    "docs-anchors": {
      "pass": 249,
      "fail": 0,
      "skip": 0
    }
  },
  "session_kind": "unattended",
  "pregate": "requires-remote cleared @6a22f8dad079 (1 checks, current for 2724f30ee090)",
  "requires_remote": [
    "free-tests.yml/macos-named-regressions: https://github.com/garrytan/gstack/actions/runs/40353300705",
    "windows-free-tests.yml/windows-free-shard: https://github.com/garrytan/gstack/actions/runs/40353300337",
    "pregate.json/remote_gate: remote-gate tree=6a22f8dad079 tests=9627"
  ],
  "issues": {
    "fixes": [],
    "refs": [
      3478
    ],
    "flakes": []
  }
}

What changed

Trains fill at 3 READY PRs or one hour (was 5 PRs or three hours), and the train verdict's ATTEMPT: line names the cases it could not price instead of printing a partial sum as the run's cost

Trains fill at 3 READY PRs or one hour (was 5 PRs or three hours), and the train verdict's ATTEMPT: line names the cases it could not price instead of printing a partial sum as the run's cost.

Before/after as text: Trains fill at 3 READY PRs or one hour (was 5 PRs or three hours), and the train verdict's ATTEMPT: line names the cases it could not price instead of printing a partial sum as the run's cost
--- Before · before.txt · captured output
+++ After · after.txt · captured output
+$ gstack-ship-policy show-effective --policy-from head --base main    # this branch
+train={"gate_workflow":"evals.yml","free_workflow":"free-tests.yml","full_gate_check":"gstack/full-gate","record_artifact":"gstack-train-record","max_size":3,"fill_hours":1,"screen_selection":"gstack-paid","trial_reservation_usd":1.5}
-$ gstack-ship-policy show-effective --base main    # origin/main@5755601
-train={"gate_workflow":"evals.yml","free_workflow":"free-tests.yml","full_gate_check":"gstack/full-gate","record_artifact":"gstack-train-record","max_size":5,"fill_hours":3,"screen_selection":"gstack-paid","trial_reservation_usd":1.5}
 
+$ gstack-ship-queue train verdict --train <nonce>   # test/ship-queue-train-cost.test.ts fixture: 2 of 3 panels unpriced
+ATTEMPT: a1 gate run 101 GREEN cost=$0.70 known + 2 of 3 case(s) cost unknown (rule 2/2, judge 1/1)
-$ gstack-ship-queue train verdict --train 76db1d1f10c4   # 54 of 136 trials unpriced
-ATTEMPT: a1 gate run 101 GREEN cost=$24.41 (rule 2/2, judge 1/1)
 

Trains fill at 3 READY PRs or one hour after the first READY, whichever comes first. .gstack/ship-policy.json moves train.max_size 5 → 3 and train.fill_hours 3 → 1. fillDue already dispatches at either bound, so this is the live block plus a pin: test/gstack-ship-policy.test.ts asserts the values and exercises fillDue with the live block at 59 and 60 minutes and at 3 PRs. A fourth READY PR is HELD: for the next train.

The train verdict's cost agrees with the slice report, and says what it could not price. train verdict printed cost=$24.41 for train 76db1d1f10c4 while the same run's slice report said cost $32.92 known + 54 of 136 trial(s) cost unknown. The slice report was right. scripts/lib/paid-report.ts matched result records to a case only by name === id or case_id; E2E records are named by a display label (/benchmark workflow) while the JUnit case is the id (benchmark-workflow), so 54 trials got cost_usd: 0, cost_known: false and the per-case sum was short by $8.51. Now a shard that ran exactly one case owns its records whatever they are named (a failed record's failure_class, exit_reason and error come from that record too); a multi-case shard with display-labelled records keeps those trials cost-unknown rather than guessing. lib/ship-queue-train-verdict.ts's ATTEMPT: line prints cost=$<known> known + <n> of <m> case(s) cost unknown when some but not every case is priced, the bare cost=$<sum> only when every case is priced, and no cost at all when none is.

/ship additions on top of the fix: a non-behavioral HARNESS_VERSION decision for scripts/lib/paid-report.ts (no case's pass or fail changes, only a trial record's cost and failure fields), and test/ship-queue-train-cost.test.ts joins test/ship-queue-train.test.ts in scripts/lib/windows-curation.ts's exclusion list because windows-latest cannot run the #!/bin/sh gh shim it drives.

Why

The merge-train coordinator reads train verdict to report spend and left READY PRs waiting on a 5-PR fill that rarely completed inside three hours; on 2026-10-26 both numbers it read were misleading in operation. Every train cost the coordinator reported until this fix was that undercounted verdict figure (about $24 per train against about $33 real). After this change the coordinator sees a train dispatch within an hour of the first READY and a cost line that names what it could not price.

Approvals

Date Asked Exact words Outcome Covers Still pending
2026-10-26 Harness phase 2 (ship-queue gaps seen in operation) “Do it and also improve it: fix the issues all the captains have seen in this Capy project” (Garry, 2026-10-26, via GStack Minder) approved this PR and its siblings none
2026-10-26 New ship-queue command surfaces and result codes (verify --screen/--post-merge, train watch/forget, prune --stale, ready --dispatch-ci, train build --dry-run --stack-tests; QUEUE_SCREEN_RED/NOT_RUN, QUEUE_POST_MERGE_RED, QUEUE_TRAIN_STACK_RED; screen/<pr>-<head12> branches) “Accepted as designed” (GStack Minder (Captain), 2026-10-26) approved #3478, #3483, #3484 none

Source: gstack-approvals record for train-fill-and-cost (display only; not an approval citation).

Summary

  • Train fill policy: .gstack/ship-policy.json train block 3 PRs / 1 hour, pinned by test/gstack-ship-policy.test.ts; docs/ship-policy.md's fill paragraph and example block updated.
  • Cost attribution: scripts/lib/paid-report.ts attributes a single-case shard's records to that case regardless of their display name; lib/ship-queue-train-verdict.ts prints the known + unknown cost form. Pinned by test/paid-report-census.test.ts and test/ship-queue-train-cost.test.ts.
  • CI hygiene from /ship: scripts/harness-version.json non-behavioral decision for paid-report.ts; Windows exclusion for test/ship-queue-train-cost.test.ts.
  • Docs: docs/TESTING_INTERNALS.md, docs/troubleshooting.md#cost-unknown and docs/evals/census-red.md describe one-case-shard attribution and the three cost-line forms; CHANGELOG ## [Unreleased] entry (no version bump: queue_mode stamp-at-merge, the train stamps at the slot).

Test Coverage

15 code paths in the diff; 13 have a test (87% value-weighted), 12 at ★★ or better (80%), authoring gate PASS. Base control: the three claimed-red tests (test/gstack-ship-policy.test.ts fill rule, test/paid-report-census.test.ts display-labelled single-case shard, test/ship-queue-train-cost.test.ts known+unknown line) fail at origin/main and pass at HEAD.

Tests: 1455 test files before → 1455 after (0 new files; 3 existing test files extended)
Coverage: 87% value-weighted (80% at ★★ or better, 1 weakly covered path)
Test value: 0 tests written, 0 rejected by the authoring gate, 3 existing tests extended, 1 path weakly covered (weak = ★, gate-failing or unrated).

  • Regression proof — test/gstack-ship-policy.test.ts live fill rule: fails at base ([5, 3]) · passes after fix.
  • Regression proof — test/paid-report-census.test.ts display-labelled record in a one-case shard: fails at base (cost_usd: 0, cost_known: false) · passes after fix; a failed display-labelled trial keeps the record's failure_class/exit_reason/error (added in this /ship, ★★★).
  • Regression proof — test/ship-queue-train-cost.test.ts known+unknown line: fails at base (bare cost=$0.70) · passes after fix; the all-unknown record prints cost=$0.00 known + 3 of 3 case(s) cost unknown only from a priced-zero record (added in this /ship, ★★).
  • Weak path (★): lib/ship-queue-train-verdict.ts costText with a.cost_usd === null prints no cost text; covered only by test/ship-queue-train.test.ts:686. Proposed test: a train record whose panels are all cost-unknown asserts the ATTEMPT: line carries no cost= token (value: pins the "no priced case prints no cost" rule the docs now state).

Tests ran: focused 341
Tests ran: guards 75
Tests ran: windows-curation 122
Tests ran: docs-anchors 249
Tests ran: typecheck count unavailable

Pre-Landing Review

Three review passes (cap, crew rule 23), plus two adversarial passes. Pass 1 (whole diff at d5e6a7d): 3 INFORMATIONAL, 0 CRITICAL, quality 9. The coverage audit found an undeclared behavior change (the widened record filter also feeds failedRecord, so a failed trial's failure_class/exit_reason/error come from the record) → fixed in the CHANGELOG and pinned by a test. Pass 2 (delta, de5d385): clean. Pass 3 (delta, bd9c934): clean. Adversarial pass at de5d385: no blocker; led to the reworded nonBehavioral note in scripts/harness-version.json (bd9c934). Adversarial pass at bd9c934: no blocker; one wording finding below.

Advisory and informational items (listed, not fixed, per the owner's blocking floor):

  • scripts/lib/paid-report.ts:280 missing-edge-case-coverage (INFO, conf 5): a single-case shard accepts any record without case_id, including one named for a different case id. Proposed test in test/paid-report-census.test.ts; not added.
  • docs/ship-policy.md:444 stale-comments (INFO, conf 6): a dated aside ("it was 5 PRs or three hours until 2026-10-26…") will age. Pre-existing prose style, deferred.
  • lib/ship-queue-train-verdict.ts:86 completeness-gaps (INFO): costText has no explicit branch for the all-unknown record (it prints nothing, which the docs now state).
  • Wording: the recorded nonBehavioral note says a red whose record says infra or timeout is classed by that record; scripts/train-gate.ts classes infra as INFRA but not timeout. The note's claim about pass/fail is unaffected.

Gate edits: 4 listed, 0 recognised gate edits (gstack-gate-diff 5755601+6a22f8dad079, coverage RH-1,RH-3,RH-4,RH-12,RH-13,RH-14,RH-15,RH-16). Self-declared gate-adjacent edits, both kept:

  • harness-version guard, scripts/harness-version.json: non-behavioral decision for scripts/lib/paid-report.ts, kept. Reason: no case's pass or fail changes; only a trial record's cost and failure fields do (actor: ship, parent-directed).
  • windows-curation subset, scripts/lib/windows-curation.ts: test/ship-queue-train-cost.test.ts excluded from the Windows subset, kept. Reason: it drives bin/gstack-ship-queue against the #!/bin/sh gh shim, which windows-latest resolves to the runner's gh.exe (windows-free-shard 2 at d5e6a7d), same reason as the existing test/ship-queue-train.test.ts entry (actor: ship, parent-directed).

Outside review: Codex unavailable in this harness (outside_status: unavailable); the adversarial passes ran in-host. Listed as missing outside coverage.

Exploratory QA

Ship smoke, report-only, 3 of 12 probes used (6.9 s guarded), verdict pass: gstack-ship-policy validate --policy-from head → POLICY_VALID; gstack-ship-policy show-effective --policy-from head --base main → max_size 3, fill_hours 1; bun test test/ship-queue-train-cost.test.ts test/paid-report-census.test.ts → 13 pass 0 fail. No findings reproduced. Not probed: a single-case shard with a foreign-named record (advisory above). Report under .gstack/qa-reports/ship-3478/ (git-excluded).

Design Review

No frontend files changed — design review skipped.

Eval Results

No paid eval dispatched: no skill prompt or template changed and no paid case reads the changed files (scripts/lib/paid-report.ts is the harness that reads results; its fixture tests are the proof). The free tier runs on the pregate's remote lane; the merge train runs the full gate at the slot.

Greptile Review

No Greptile comments (GREPTILE_EARLY: off; fetch succeeded, only trunk-io[bot] had commented).

Plan Completion

No saved plan file for this branch; Step 8 plan completion not applicable.

Issues

Refs #3478
reference #3478 ← commit : test/ship-queue-train-cost.test.ts drives bin/gstack-ship-queue against the #!/bin/sh gh shim, which windows-latest cannot run (windows-free-shard 2 on #3478 at d5e6a7d), so it joins test/ship-queue-train.test.ts in the Windows exclusion list. (promote: fix #3478)

Verification Results

Step 8.1 obligations: none (no plan file). Contracts verified at Step 9 by QA smoke: 3 PASS, 0 FAIL, 0 BLOCKED, 0 NOT RUN.

TODOS

No TODO items completed in this PR.

Documentation

Status: updated — second audit (ship-3478-docs-reaudit-153813) verified the first audit's three doc edits against junitCensus, trialCostKnown, train-gate.ts, costText and fillDue, found the new CHANGELOG sentence consistent with every doc, and tightened one rule in two docs: a train verdict's ATTEMPT: line prints cost=$<known> known + <n> of <m> case(s) cost unknown when some but not every case is priced, and no cost at all when no case is.

Audited scope. Base 5755601 → head 1b3aab9 on train-fill-and-cost (5 commits, 13 files), clean tree: .gstack/ship-policy.json (train 3 PRs / 1 hour), lib/ship-queue-train-verdict.ts (costText), scripts/lib/paid-report.ts (one-case-shard record attribution), scripts/lib/windows-curation.ts, scripts/harness-version.json, three tests, CHANGELOG ## [Unreleased], and the docs below. 123 documentation files inventoried; the four touched by the diff plus README, CONTRIBUTING, CLAUDE.md and AGENTS.md read against the diff. Banned terms: none.

Documentation health:
docs/TESTING_INTERNALS.md Updated (ATTEMPT: cost line prints the known+unknown form when some but not every case is priced; a record with no priced case prints no cost)
docs/troubleshooting.md Updated (#cost-unknown: bare cost=$<x> only when every case is priced, no cost at all when none is)
docs/evals/census-red.md Current (cost unknown row matches junitCensus and trialCostKnown)
docs/ship-policy.md Current (fill rule, 3 PRs / one hour, example block and QUEUE_TRAIN_WAITING match fillDue and .gstack/ship-policy.json)
CHANGELOG.md Current (not mutated; its Fixed bullet says the known+unknown line prints "when any panel has no cost", which glosses the all-unknown record that prints nothing)
README.md / CONTRIBUTING.md / CLAUDE.md / AGENTS.md Current (no train fill or cost attribution content; none needed)

Documentation coverage:
train.max_size 3 / fill_hours 1 ✅ reference (ship-policy.md table, example block) ✅ how-to (run-a-merge-train) ❌ tutorial ✅ explanation (fill paragraph)
one-case-shard cost attribution ✅ reference (TESTING_INTERNALS, census-red) ✅ how-to (troubleshooting) ❌ tutorial ✅ explanation (TESTING_INTERNALS)
ATTEMPT: cost=$ known + … ✅ reference ✅ how-to (troubleshooting) ❌ tutorial ✅ explanation

Coverage debt: no tutorial covers merge trains or paid cost reporting (pre-existing, common gap, not critical). Diagram drift: none; no diagram entity was renamed, split, removed or moved in this diff. The audit's two doc edits and the matching CHANGELOG wording are committed in 2724f30.

Test plan

  • bun test test/paid-report-census.test.ts test/paid-report-fail-open.test.ts test/train-gate.test.ts test/gstack-ship-policy.test.ts test/ship-queue-train-cost.test.ts test/ship-queue-train-logic.test.ts test/module-size-ratchet.test.ts test/troubleshooting-anchors.test.ts test/harness-version.test.ts (focused): 341 pass, 0 fail
  • bun test test/parity-suite.test.ts test/bun-entry-isolation.test.ts test/spawnsync-timeout-tripwire.test.ts test/ship-queue-train.test.ts test/evals-train-workflow.test.ts test/changelog-merge-damage.test.ts (guards): 75 pass, 0 fail
  • bun test test/test-free-shards.test.ts test/windows-native-workflows.test.ts test/setup-hook-syntax-gate.test.ts test/ubicloud-pregate-remote.test.ts (windows-curation): 122 pass, 0 fail
  • bun test test/cost-known.test.ts test/census-red-docs.test.ts test/testing-internals-anchors.test.ts test/troubleshooting-anchors.test.ts (docs-anchors): 249 pass, 0 fail
  • bun run typecheck: exit 0
  • gstack-pregate --base main --allow-repo-commands: preflight 7 checks pass; tests tier requires-remote on this 4-core machine, cleared by the remote Ubicloud free-tier lane (run linked in the READY report)
  • bun run test locally: not run (4-core machine; the remote lane runs the full free tier)

🤖 Generated with gstack /ship under Capy

…ery case shard's trial

The 5-PR / 3-hour fill left READY PRs waiting most of a day; .gstack/ship-policy.json
now fills at max_size 3 or fill_hours 1, whichever comes first (fillDue already
dispatches at either bound). train verdict printed cost=$24.41 for a train whose slice
report said cost $32.92 known + 54 of 136 trial(s) cost unknown: E2E records named by a
display label ("/benchmark workflow") were never attributed to their case id, so the
per-case sum was short. A case shard's records now belong to its one case; a multi-case
shard with display-labelled records keeps those trials unknown. The ATTEMPT: line names
how many cases have no cost instead of printing a partial sum as the total.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

garrytan and others added 5 commits October 26, 2026 15:17
…ue-train-cost.test.ts off the Windows subset

scripts/lib/paid-report.ts is a pinned shared-harness file, so test/harness-version.test.ts wants a HARNESS_VERSION decision for its edit: non-behavioral, because a case shard's records are attributed to its one case for the trial's cost, class and error fields while no case's outcome, panel verdict or selection changes.

test/ship-queue-train-cost.test.ts drives bin/gstack-ship-queue against the #!/bin/sh gh shim, which windows-latest cannot run (windows-free-shard 2 on #3478 at d5e6a7d), so it joins test/ship-queue-train.test.ts in the Windows exclusion list.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
… line; say so in the CHANGELOG

The widened record filter also feeds failedRecord, so a failed display-labelled trial in a case shard now takes failure_class, exit_reason and error from its record (train-gate classes a red infra record INFRA instead of VERDICT). The CHANGELOG entry said only the cost fields changed; it now says both. A train priced at zero prints cost=$0.00.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
The non-behavioral note now names the one verdict-side effect of attributing a case shard's records to its case: a red whose own record says infra or timeout is classed INFRA instead of VERDICT, which changes rerun and bisect eligibility and not a pass rate.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…cost line; put the undercounted train costs on record

/document-release (ship-owned audit) added the attribution rule and the ATTEMPT: cost-line form to docs/TESTING_INTERNALS.md, docs/troubleshooting.md#cost-unknown and docs/evals/census-red.md. The CHANGELOG headline now says every train cost the coordinator reported until this fix was the undercounted verdict figure (about $24 per train against about $33 real).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…some but not every case is priced

The second /document-release audit tightened docs/TESTING_INTERNALS.md and docs/troubleshooting.md#cost-unknown: a train record with no priced case prints no cost at all (classifyGate's cost is null). The CHANGELOG bullet says the same.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
garrytan and others added 2 commits October 26, 2026 20:19
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

# Conflicts:
#	CHANGELOG.md
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@github-actions github-actions Bot changed the title fix(ship-queue): fill trains at 3 READY PRs or one hour, and price every case shard's trial v1.91.255.0 fix(ship-queue): fill trains at 3 READY PRs or one hour, and price every case shard's trial Oct 26, 2026
@capy-ai
capy-ai Bot merged commit b004c06 into main Oct 26, 2026
62 checks passed
garrytan added a commit that referenced this pull request Oct 26, 2026
Rule 19 grant (train 15 QUEUE_TRAIN_CONFLICT). CHANGELOG additive. test/fixtures/context-budget.json by hand: canary keeps the hoisted ceiling (main did not change it; the conflict was main's adjacent new captain row); captain (new on main, now hoisted) and five loaded-skill ceilings lowered to a fresh capture (captain 12011 -> 6551, autoplan, office-hours, plan-ceo-review, plan-devex-review, plan-eng-review by 2-22). No ceiling raised. Generated files and ship goldens regenerated.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant