Skip to content

Plan a trial of /ship without its separate conductor subagent (GST-3003) - #4456

Merged
capy-ai[bot] merged 3 commits into
mainfrom
capy/plan-a-trial-of
Nov 17, 2026
Merged

capy-ai[bot] merged 3 commits into
mainfrom
capy/plan-a-trial-of

Conversation

@garrytan

@garrytan garrytan commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

Adds the locked plan for a paid trial: does /ship cost less and catch as much when the pull request's own thread runs it, instead of a separate subagent?
Why: that subagent's own steps cost about $22 of a $35 /ship run, and the keep-or-drop decision is written down here before anything is spent.
Nothing to do: no code changes; paid runs start only after this merges and the budget is approved.

What changed

Catch rule (b) now credits a fix only when the sealed regression test passes, so a reworded but unfixed bug earns nothing

Catch rule (b) now credits a fix only when the sealed regression test passes, so a reworded but unfixed bug earns nothing.

Before/after as text: Catch rule (b) now credits a fix only when the sealed regression test passes, so a reworded but unfixed bug earns nothing
--- Before · docs/evals/ship-conductor-fold-prereg.md § Scoring at 0ee5914 · source text (new in this change)
+++ After · docs/evals/ship-conductor-fold-prereg.md § Scoring on this branch · source text
+## Scoring
+judge: gpt-6-astra
+labels: 48
+agreement: 0.6
+paired: yes
+interval: 95%
+mdd: about $7 a run on cost (20% of C's $35; estimated from the profile's 121 to 308 conductor steps, so a per-run spread near $8, and 3 trials a case), and about 35 percentage points on catch rate; the pilot's measured spread is printed beside these and the bars do not move
+
+**Catches, by rule (no judge).** A trial catches its planted bug when either (a) a finding in the run's findings
+record (`merged.jsonl`) or final report names the labelled path, a line within 3 of the labelled line, and at least
+one cause word; or (b) the sealed regression test for the plant (from the label file, run by the runner, never shown to
+the arm) passes at the run's final head, and one of the run's findings or commit messages names that file. A case's catch rate in an arm is its catches over 3 trials. The script that applies this
+rule is checked against the decoys below before any trial runs.
+
+**Real problems, by judge.** Every finding that is not the plant and that the run marked blocking, or that led to a
+fix, goes to GPT-6 Astra with the case diff. It answers two questions: does this meet the blocking floor (would fail
+a test, break production, or lose data or secrets), and which other findings name the same problem. A case's real
+problems are the union of judged-real problems across all arms and trials; each arm's catch rate on them uses the
+rule above, with the judge's grouping in place of the planted label. The judge sees findings as A, B and C, never an
+arm name, model or cost. It is checked first against 48 hand-checked labels: findings from merged gstack pull
+requests outside the case set, 24 that led to a fix and 24 dismissed below the floor, each read against its diff by
+the runner and frozen with a SHA-256 before the judge sees any. `gstack-eval-plan judge --check` must print
+`JUDGE_AGREEMENT: ok` (Cohen's kappa at least 0.6) before its numbers count.
+
+**Comparisons.** Every comparison is paired by case: the per-case difference of means (cost, wall time, `calls=`)
+or of catch rates, with a 95% t interval over the 10 cases. Every trial is reported; nothing is averaged away.

Today every /ship run on Capy starts a separate subagent just to drive /ship's phases, and a profile of 12 runs put that subagent's own steps at most of the run's cost. This plan fixes, before any money is spent, how we will test dropping it. It uses three arms on the same 10 real gstack pull requests (today's /ship, /ship run by the thread itself, and the same model with no gstack and a plain review prompt), 3 trials each, paired comparisons with 95% intervals, the decision for every outcome, and stop rules under a $3,250 cap. Review caught one real hole before freeze, because a reworded but unfixed planted bug could have counted as fixed. The scoring rule now requires the sealed regression test to pass.

Why

"Does /ship need its conductor subagent?" (GST-3003). Writing the bars, arms and decisions down first means the result decides the question instead of being read to fit it.

Blast radius

One new Markdown file under docs/evals/ and one changelog fragment; no code, prompt or skill changes. gstack-docs-only classes the file as pinned results, so the full CI suites still run (script ran: DOCS_ONLY: no … pinned results), and gstack-eval-plan prereg --check prints PREREG: ok present=8/8.

Pre-Landing Review

Light review (docs class, 0 runtime lines). Pass 1: my checklist read plus a second-model review (GPT-6.1 Sol), which both found the same problem: catch rule (b) credited a reworded but unfixed planted bug, contradicting the cosmetic-edit decoy. The plan's author accepted it and the rule was amended (commit 70933db). Pass 2 checked the fix with a fresh reviewer and a red-team reader, and found nothing blocking. Six notes below the floor stay open for the author: the label file should list the sealed regression test so its posted hash covers it; real problems have no sealed test, so say they are scored by clause (a) only; no decoy exercises clause (b)'s credit path (add a test-passes decoy and require the runner to prove the test ran); and a source fix's own test may still sit in the arm's checkout.

Review notes

Below the blocking floor (advisory, informational, test-gap or wording items); recorded here and not fixed in this PR.

  • INFORMATIONAL docs/evals/ship-conductor-fold-prereg.md:95 red-team
  • INFORMATIONAL docs/evals/ship-conductor-fold-prereg.md:127 red-team
  • INFORMATIONAL docs/evals/ship-conductor-fold-prereg.md:140 red-team
  • INFORMATIONAL docs/evals/ship-conductor-fold-prereg.md:96 fix review
  • INFORMATIONAL docs/evals/ship-conductor-fold-prereg.md:140 fix review
  • INFORMATIONAL docs/evals/ship-conductor-fold-prereg.md:120 fix review
Exploratory QA

Functional smoke, 4 probes, verdict pass: gstack-eval-plan prereg --check (PREREG: ok), gstack-docs-only --base origin/main (DOCS_ONLY: no, pinned results), and the repository references the plan relies on (commit 0ee5914, ship/DRIVER.md, bin/gstack-ship-phase, eval-plan judge and results checks, voice-models.json as a per-pass file from lib/review-pass.ts) all resolve.

Design Review

No frontend files changed — design review skipped.

Eval Results

No prompt-related files changed — evals skipped.

Scope Drift

Scope Check: CLEAN

Verification Results

No plan file; no plan-specific verification items. Automatic QA above.

TODOS

No TODO items completed in this PR.

Documentation

Status: current. No documentation changes were needed.

Scope: base 0ee5914 to head 70933db on capy/plan-a-trial-of. The selected content is the committed docs/evals/ship-conductor-fold-prereg.md and the untracked changelog.d/capy-plan-a-trial-of.md. Nothing is staged or unstaged. The audit read both files in full, checked their cross-references in docs/inventory.md and docs/review-policy.md, and checked the commands and markers the prereg names against bin/gstack-ship-phase, lib/ship-clock.ts, lib/ship-budget-cli.ts, lib/ship-call-log.ts and lib/review-pass.ts.

Documentation health:

  • docs/evals/ship-conductor-fold-prereg.md: Current. It is new, and its factual claims match the code at the base commit.
  • changelog.d/capy-plan-a-trial-of.md: Current. It was reviewed only, because the parent owns changelog writes.

Coverage debt: none. The change is a preregistration with no shipped behavior to document. The results doc it names, docs/evals/ship-conductor-fold-results.md, is expected only after the paid run. No diagrams are affected.

Tests
  • Targeted tests (eval-plan, inventory and docs-only tests): 120 pass, 0 fail
  • Full free suite (bun run test): carried from CI run 44137100344 at 80e9f70, 37,510 tests in 1,681 files, 0 fail
Agent notes

Refs GST-3003

Approvals

No approvals recorded for branch capy/plan-a-trial-of.
Plan: required
Plan check: PLAN_NOT_APPROVED (gstack-approvals plan-check says why)

Gate edits: none detected (0 listed, 0 read; patterns: RH-1,RH-3,RH-4,RH-12,RH-13,RH-14,RH-15,RH-16; unlisted idioms: none)
Outside review: openai/gpt-6.1-sol (host subagent), completed, 1 finding (fixed)

{
  "schema_version": 1,
  "pr": 4456,
  "head": "80e9f70014defca385380985aa0413a3d02a5178",
  "version": "1.100.0.0",
  "base": "main",
  "gate": {
    "targeted": {
      "pass": 120,
      "fail": 0,
      "skip": 0
    },
    "test": {
      "pass": 37510,
      "fail": 0,
      "skip": 0
    }
  },
  "ci_carried": [
    {
      "lane": "test",
      "run": 44137100344,
      "attempt": 1,
      "tree": "41c40cd4dcd09cbc9cc7ed8f99738333851f025e"
    }
  ],
  "calls": 24,
  "session_kind": "unattended",
  "run": "ship-4456",
  "head_tree": "41c40cd4dcd09cbc9cc7ed8f99738333851f025e",
  "product_patch_id": "9639690b2dfeea12042ba7f86f42d39dbe90b8ff",
  "gstack_version": "1.100.0.0",
  "review_depth": {
    "depth": "light",
    "class": "docs",
    "runtime_lines": 0,
    "rules": "66e5174161eecc2c",
    "base": "0ee5914628eb0373876bfc9c511a48178a6700fa",
    "registry": "0ee5914628eb0373876bfc9c511a48178a6700fa",
    "gate_scan": {
      "tagged": 0,
      "head": "80e9f70014defca385380985aa0413a3d02a5178"
    },
    "reasons": [
      "runtime=0"
    ],
    "risk": [],
    "mode": "panel"
  },
  "pregate": "pass @41c40cd4dcd0 (8 checks, current for 80e9f70014de)",
  "issues": {
    "fixes": [],
    "refs": [],
    "flakes": []
  },
  "docs": {
    "status": "current",
    "scope": "docs/ and README.md: the new preregistration and its changelog fragment, cross-checked against docs/inventory.md, docs/review-policy.md and the bins it names"
  },
  "review_loop": {
    "passes": 2,
    "blocking_found": 1,
    "deferred": 0,
    "noted": 6,
    "fixed": 1,
    "scope": "frozen",
    "fix_checks": 1,
    "rounds": 3,
    "ledgers_read": 3,
    "unbound": 1
  },
  "gate_dispositions": []
}

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

garrytan and others added 2 commits November 17, 2026 17:10
…T-3003)

Review-Finding: p1/outside-voice#1 reader=outside-voice
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@capy-ai
capy-ai Bot merged commit 88da124 into main Nov 17, 2026
77 checks passed
This was referenced Nov 18, 2026
capy-ai Bot added a commit that referenced this pull request Nov 18, 2026
- A test, or a scratch repository in the temp directory, can no longer write approval rows into your real Capy Drive (GST-3027). (#4466)
- An unattended /autoplan run on Claude Code no longer says the publication guard is off while it is on. (#4481)
- A phase report written only in reasoning now gets a denial that says so and names the fix. (#4481)
- `gstack-autoplan revise` no longer refuses a plan that the first `next` accepted. (#4470)
- `revise --plan` with a different file now says the run follows that file. (#4470)
- A second-model plan review in `/autoplan` is no longer reported as unavailable when it actually ran (GST-3051). (#4471)
- The /ship conductor trial plan now lets case reverts that conflict with later code be resolved by hand in docs, tests and the planted file, so large pull requests can be cases (GST-3003). (#4460)
- A locked plan for a trial of /ship without its separate conductor subagent (GST-3003). (#4456)
- A driven /ship now runs each phase as one command, counts its calls, and stops itself at 100 minutes. (#4451)
- /ship stops asking owners to clear two kinds of bookkeeping noise: base-side merge text and a reviewer's confirmation prose. (#4458)
- A base merge that only touched base-side text no longer stops `/ship` finish for an owner's grant. (#4458)
- A reviewer's confirmation prose no longer becomes a Critical finding that only an owner can clear. (#4458)
- A CI run that passed only after its failed jobs were re-run is now marked wherever gstack reuses it. (#4473)
- /ship reuses CI's passing test run when merging the base changed no file, and stops instead of quietly starting hour-long test runs or reviewing an empty diff (GST-3019). (#4461)
- /ship on Codex no longer stops at the test coverage audit. (#4489)
- A cold first-use check before an app build reaches its owner (GST-3025). (#4465)
- `skill_prefix=true` now survives updates when gbrain is installed. (#4454)
- The OpenClaw docs no longer send you to ClawHub, which held only April 2026 builds of office-hours, ceo-review and retro and no longer has gstack skills (the docs also said four skills while listing three). (#4454)
- `gstack-issue-links` no longer reads other repositories' issues as gstack's. (#4454)
- /ship's pre-gate finds the tests of a monorepo app that imports through its own `@/` alias. (#4464)
- /office-hours explains why it recommends a mode. (#4450)
- /office-hours keeps its 34 founder videos and essays out of the skill file. (#4463)
- Two `gstack-owner-answers add` runs for the same answer no longer collide. (#4490)
- A Captain can check an ask against the owner's earlier answers before sending it. (#4467)
- `gstack-post` stops reading /ship's own receipt identifiers as phone numbers, so posting a /ship PR body no longer needs a confirm. (#4488)
- GitHub release notes now list every change in plain words, with a link to its pull request. (#4390)
- gstack's router no longer gives bug reports a special rule, so the model fixes a bug the way it would without gstack. (#4442)
- /ship's review plan now sorts Swift and Xcode files by risk instead of sending every iOS PR to the full review panel (GST-3041). (#4469)
- /ship's review plan now sends Swift code that touches private user data to the full review panel (GST-3052). (#4472)
- A round grant at /ship's review round cap now buys the fix round it names, in the same run or the next (GST-2968). (#4479)
- The secrets check stops flagging three kinds of ordinary code as a secret, and catches two misses. (#4457)
- A security finding's location and reason no longer go into a PR body. (#4486)
- The token-capped /ship child on Claude Code now starts its prompt with the typed `/ship` command, so the skill actually loads. (#4487)
- A /ship receipt now counts the second-model review's blocking findings, and refuses to be written while one is still open. (#4485)
- A `/ship` with a token cap on Claude Code no longer stops at its first handoff. (#4452)
- /ship's push step now always ends with one line that says whether it pushed (GST-3053). (#4474)
- A PR's review-round count can no longer be overwritten by the body `/ship` posts. (#4440)
- One queued record with a malformed path no longer stops artifacts sync for every project. (#4455)
- A failed artifacts push stays reported until a push succeeds. (#4455)
- The /sync-gbrain memory row no longer says OK when pages failed to import. (#4455)
- ### Tests (#4495)
- A policy-floor test no longer fails once gstack's own version reaches the minor after the newest policy key; it now checks a floor above both, so the v1.101.0.0 release can pass CI. (#4495)
- CI now uploads free-suite test results to Trunk with analytics-uploader v2.1.8 and analytics-cli 0.17.0, up from v2.1.6 and 0.16.2. (#4453)
- Packing a checkout for a Ubicloud run no longer fails when git beside it is rewriting a ref. (#4447)
- The /plan-ceo-review HOLD SCOPE eval no longer fails a review because its follow-up question didn't repeat the mode name. (#4447)
- Unattended /autoplan now records the owner's relayed gate reply instead of refusing it. (#4484)
- Unattended /autoplan now runs the second-model review you turned on, and waits for it instead of ending the session. (#4459)
- Two /autoplan runs started in the same second no longer share a run id. (#4459)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant