Skip to content

Second-model review keeps the reviewer's result and reports budget stops (GST-3181) - #4515

Merged
capy-ai[bot] merged 5 commits into
mainfrom
report-review-budget-stop-not-login
Nov 19, 2026
Merged

capy-ai[bot] merged 5 commits into
mainfrom
report-review-budget-stop-not-login

Conversation

@garrytan

@garrytan garrytan commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

When Codex reviews a change, gstack now keeps the review file the reviewer wrote instead of replacing it with Codex's one-line closing message, and a provider that refuses with HTTP 402 is reported as a used-up budget instead of "exit 1" or a login failure.
Why: gstack-evals lost 22 review attempts to the overwrite and 5 of 45 sessions to a 402 from a metering proxy whose cause gstack threw away.
Nothing to do: a 401 or 403 is still a login problem. Open notes, one of them a known gap, are listed below.

What changed
  • The reviewer's result survives. The autoplan prompt tells the reviewer to write its result to the file gstack passes as --out, and the codex-cli runner also passed that path to codex exec -o, so Codex overwrote the review with its closing message when it exited. -o now writes <out>.last-message; gstack keeps the reviewer's own file when it carries the gate's markers, otherwise the last message (the old behavior), and writes the winner to <out>. Confirmed with codex-cli 0.170.0.
  • A 402 keeps its cause. Real Codex prints no stderr line for an HTTP refusal; the cause is only in its --json error and turn.failed events. The shared validator (lib/outside-review-result.ts) now reads those events, and a 402 or "payment required" becomes the new reason budget_exhausted (lib/gate-outcomes.ts, anchor outside-review-budget-exhausted in docs/troubleshooting.md). Other failures, 401 and 403 included, keep Codex's status line as their detail instead of "exit 1".
  • Every path. The same validator serves the rendered /review, /ship and plan-review calls, the /codex modes, the unattended runner and host-dispatched reviewers. The Claude Code review that a Codex session runs gets a budget failure code in lib/claude-code.ts. /codex's own error list names the budget stop before its auth error. The prose auth rule the other skills share is unchanged (see the open notes).
Not changed

When the review runs, retries, budgets and allowances are unchanged. The readiness probe still treats a 402 as inconclusive, so the review runs once and its own result names the cause. A reviewer in a read-only sandbox still cannot write its own file; the last message is used then, as before.

Open notes (not fixed in this PR)
  • Still open from the original request: the prose rule the skills share still reads stderr containing "auth", "login" or "API key" as an auth failure. So in every skill except /codex, a 402 whose provider message mentions the API key can still be read as a login problem. This is the one part of the request this PR doesn't close.
  • Ruled below the blocking bar: lib/outside-voice-runner.ts:158. The runner keeps the reviewer's own file unless that file is empty or missing its markers, so an incomplete draft without a final verdict wins over a finished closing message in .last-message. No current caller reaches this (only the autoplan prompts ask the reviewer to write its own file, and the autoplan gate reads an incomplete draft as missing its markers), and it fails closed: the run reads as unverified, never as clean. Follow-up (two lines): count the reviewer's own file only when its verdict is clean or findings.
  • lib/outside-review-result.ts:69: the 402 pattern matches any "402" with no digit beside it, so a request id like 9c3f402ab or "took 402 ms" reads as a budget stop. A 401 or 500 with such an id is reported as budget_exhausted and tells the user logging in won't help (reproduced). The review still reads as unavailable. Follow-up: match 402 only as an HTTP status ("status 402", "HTTP 402", "Payment Required") and add negative tests.
  • lib/outside-review-result.ts:117: a budget or quota line in the host's error text is checked before the review itself, so a stale host error from a retry that later recovered can discard a finished review.
  • lib/claude-code.ts:122: a Claude Code usage limit is reported with the Codex usage-limit message, which names Codex and a Codex-only setting.
  • lib/claude-code.ts:118: when stdout is JSON, the budget check reads only the result text, so a 402 Claude Code put only in another field falls through to the generic failure.
  • lib/outside-voice-runner.ts:159: with the Codex sandbox turned off (GSTACK_CODEX_NO_SANDBOX=1), code the reviewer runs could write a marked result file that now beats Codex's real closing message. The default read-only sandbox can't write that file.
  • lib/outside-voice-runner.ts:145: the .last-message copy is left beside the .stderr and .events.jsonl files after the run.
  • Ruled below the blocking bar (maintainer ruling, 2026-11-19), from the second-family read of the decision trail. Each one fails toward a wrong or generic cause, never toward a pass. A follow-up PR will cover the first three:
    • lib/outside-review-result.ts:128: the first matching failure line wins, so an earlier Codex error event with a 402 followed by a final turn.failed with a 401 reads as budget_exhausted.
    • lib/claude-code.ts:116: Claude Code exiting 0 with subtype: error_during_execution, a 402 in result and no is_error skips budget detection and reports the generic provider-error.
    • lib/outside-voice-runner.ts:148: the runner deletes <out>.last-message before Codex runs without checking it isn't the prompt file, so --prompt X.last-message --out X deletes the prompt file after reading it. No caller passes that pair.
    • lib/outside-voice-runner.ts:158 (same spot as the ruling above): a well-formed reviewer file with the wrong snapshot hash beats a correct .last-message, and autoplan then rejects it. This fails closed.
Documentation

Status: current. Docs match the branch's changes; no corrections needed (read-only audit).

Scope: base b702243 to head ec512c3 on report-review-budget-stop-not-login: committed changes only, as the tree is clean.

Documentation health

  • docs/troubleshooting.md: Current (new outside-review-budget-exhausted entry matches budget_exhausted in lib/gate-outcomes.ts)
  • codex/SKILL.md.tmpl: Current (402 budget rule; auth rule limited to 401/403)
  • codex/SKILL.md: Current (in sync with template)
  • changelog.d/report-review-budget-stop-not-login.md: Current (matches the .last-message and 402 behavior in code)

Debt: no coverage gaps and no diagram drift.

Tests
  • Focused tests (claude-code-runner, model-catalog-isolation, outside-review-result, outside-runner-contract, outside-voice-failure-policy, outside-voice-invocation) all pass at ec512c3, with no failures.
  • bun run typecheck: clean. The parity suite passes.
  • Pre-push checks: typecheck, skill size, parity suite, context budget, module size, goldens and conflict markers passed at ec512c3.
  • Full free suite: carried from CI run 44494151996 at ec512c3 (1702 files).
  • Paid evals: none run for this PR; the selector's 106 behavioral probes ride the scheduled release coverage.
Agent notes

Lane GSTA-642 for GStack Minder. /ship run ship-4515-ec512c3: one read-only full review of the final head ec512c3 (restores main's auth-rule prose byte for byte and adds the three new imports to test/model-catalog-isolation.test.ts), no fix round, per the Captain's ruling A1. Earlier /ship ship-4515-07967db reviewed 07967db, then fix-checked 042eee7.

REVIEW_LOOP: passes=1 blocking_found=1 deferred=0 noted=4 fixed=0 scope=frozen open=1 ledgers_read=2 dispatches=2 est_input_tokens=22001
Gate edits: none detected (GATE_SCAN tagged=0 at ec512c3)
Ruled note: lib/outside-voice-runner.ts:158:outside-voice at 042eee7 ruled by Captain GStack Minder (GSTA-36) at 2026-11-19T08:09:18.554Z: no current caller reaches it (only autoplan prompts ask for the reviewer file, and the autoplan gate reads an incomplete draft as missing_markers) and it fails closed to unverified (approval a-b3a6a1209d5e)

Review: full panel (gate class, money risk), round 2 of 2. Batched specialists (testing, security, state-machines), red team, native adversarial, second-model review on openai/gpt-6.1-sol (completed), targeted money reader (NO FINDINGS). The one Blocks: yes finding is outside-voice#1 at lib/outside-voice-runner.ts:158, unchanged since 042eee7 and ruled below the floor. Everything else is informational and listed under Open notes.

Approvals

Approved: yes (record r-bb6052beaded). The full approval record stays private.
Plan: exempt: bug fix
Plan check: exempt bug fix

{
  "schema_version": 1,
  "pr": 4515,
  "head": "ec512c31ba6ddafeb380b49e85cccf6f0486e802",
  "version": "1.101.0.0",
  "base": "main",
  "gate": {
    "test": {
      "pass": 1702,
      "fail": 0,
      "skip": 0
    }
  },
  "ci_carried": [
    {
      "lane": "test",
      "run": 44494151996,
      "attempt": 1,
      "tree": "b0ad51908e135353c4d1ae7fe14163828b356c5c"
    }
  ],
  "calls": 34,
  "session_kind": "unattended",
  "run": "ship-4515-ec512c3",
  "head_tree": "b0ad51908e135353c4d1ae7fe14163828b356c5c",
  "product_patch_id": "1683b5bfd154e74e9f431a1851deb5c72bb98f59",
  "gstack_version": "1.101.0.0",
  "review_depth": {
    "depth": "full",
    "class": "gate",
    "runtime_lines": 61,
    "rules": "1afa38a4be3b5eef",
    "base": "f13dbb0d4a3e033cd086eadc8aed2ee1c91ac954",
    "registry": "f13dbb0d4a3e033cd086eadc8aed2ee1c91ac954",
    "gate_scan": {
      "tagged": 0,
      "head": "ec512c31ba6ddafeb380b49e85cccf6f0486e802"
    },
    "reasons": [
      "gate-class",
      "risk:money",
      "runtime=61"
    ],
    "risk": [
      "money"
    ],
    "mode": "panel"
  },
  "pregate": "requires-remote cleared @b0ad51908e13 (8 checks, current for ec512c31ba6d)",
  "requires_remote": [
    "free-tests.yml/macos-named-regressions: https://github.com/garrytan/gstack/actions/runs/44494151996/attempts/1",
    "windows-smoke.yml/windows-smoke: https://github.com/garrytan/gstack/actions/runs/44486803742/attempts/1",
    "pregate.json/remote_gate: remote-gate tree=b0ad51908e13 tests=37914 skipped=104 runs=44494151996/1 profiles=ci-reuse/free-tests heads=ec512c31ba6d verified=2026-11-19T09:33:24.105Z"
  ],
  "issues": {
    "fixes": [],
    "refs": [],
    "flakes": []
  },
  "docs": {
    "status": "current",
    "scope": "docs/ and README.md (read-only audit, base b7022438 to ec512c31)"
  },
  "review_loop": {
    "passes": 1,
    "blocking_found": 1,
    "deferred": 0,
    "noted": 4,
    "fixed": 0,
    "scope": "frozen",
    "rounds": 1,
    "est_input_tokens": 22001,
    "ledgers_read": 2,
    "dispatches": {
      "specialist-batched": 1,
      "red-team": 1
    }
  },
  "ruled_notes": [
    {
      "finding": "lib/outside-voice-runner.ts:158:outside-voice",
      "head": "042eee74ff6ae4aeb3d31b419ee3f2a55fe42d21",
      "by": "Captain GStack Minder (GSTA-36)",
      "reason": "no current caller reaches it (only autoplan prompts ask for the reviewer file, and the autoplan gate reads an incomplete draft as missing_markers) and it fails closed to unverified",
      "at": "2026-11-19T08:09:18.554Z",
      "approval": "a-b3a6a1209d5e"
    }
  ],
  "gate_dispositions": [],
  "review_targeted": {
    "classes": [
      "money"
    ],
    "rules": "23eb808423669ba0",
    "brief_sha": "7f92a0d0a8319c3c0e076fb3b1a4689c45120b2be2c14dfc7d4cb03aafa9d6a5",
    "product_patch_id": "1683b5bfd154e74e9f431a1851deb5c72bb98f59",
    "status": "done",
    "findings": 0,
    "raised_by": "targeted"
  }
}
{"head":"042eee74ff6ae4aeb3d31b419ee3f2a55fe42d21","run":"ship-4515-07967db","status":"started","review_loop":{"passes":1,"blocking_found":1,"deferred":0,"noted":4,"fixed":3,"scope":"frozen","fix_checks":1,"rounds":2,"dispatches":{"specialist":3,"red-team":1}}}
{"head":"ec512c31ba6ddafeb380b49e85cccf6f0486e802","version":"1.101.0.0","gstack_version":"1.101.0.0","run":"/home/user/.capy/work/gstack/ship-4515-ec512c3","review_loop":{"passes":1,"blocking_found":1,"deferred":0,"noted":4,"fixed":0,"scope":"frozen","rounds":1,"est_input_tokens":22001,"ledgers_read":2,"dispatches":{"specialist-batched":1,"red-team":1}},"depth":"full"}

…ops (GST-3181)

Codex's closing message no longer overwrites the result file the reviewer wrote, and an HTTP 402 refusal is recorded as budget_exhausted with its cause instead of exit 1 or a login failure.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

garrytan and others added 4 commits November 19, 2026 07:41
… Codex-host skills

The wrapper appends its budget line to its own diagnostic output, so the rendered invocation is unchanged; the shorter auth rule lowers 27 size pins by 2 characters.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…402 budget stop

Claude Code's budget check no longer reads the raw result JSON, where numbers such as
duration_ms 402 or a session id turned a 401 into a budget stop, and a usage limit is
no longer worded as HTTP 402. Codex's execution_failed detail now names the final
turn.failed status instead of an earlier transient reconnect message.

Review-Finding: p1/specialist-batched#1 reader=testing
Review-Finding: p1/red-team#1 reader=red-team
Review-Finding: p1/red-team#2 reader=red-team
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…n test the new imports

Restores main's auth-rule wording in the shared failure prose, so the hash-pinned review and ship briefs, the skill size pins and the ship size ledger are unchanged. The isolation test now copies outside-review-result.ts and its two imports, which claude-code.ts now loads.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@capy-ai
capy-ai Bot merged commit 957fab7 into main Nov 19, 2026
82 of 84 checks passed
capy-ai Bot added a commit that referenced this pull request Nov 19, 2026
…er deletes its own prompt (GST-3188) (#4519)

When a second-model review fails, gstack now reports the failure that
actually ended it: Codex's final error decides, a stray "402" in a
request id no longer reads as a used-up budget, and a Claude Code 402 or
usage limit reads as a budget stop with no Codex advice.

Why: these were the four follow-ups ruled below the blocking bar on
#4515; each one could send someone to fix the wrong thing.

Nothing to do. The prose rule that reads "auth", "login" or "API key" in
stderr as a login failure is not changed here (see below).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@garrytan garrytan mentioned this pull request Nov 19, 2026
capy-ai Bot added a commit that referenced this pull request Nov 19, 2026
- `gstack-artifact validate` no longer passes a run.json that gstack-autoplan never wrote. (#4494)
- /ship's changed-number check no longer stops on tests that only share a word with the change (#4478). (#4496)
- On Capy, /ship no longer edits or comments on pull requests as you. (#4518)
- The destructive-command check stops asking about scratch deletes and catches the real damage it used to miss. (#4513)
- The destructive-command check reads commands more carefully, closing gaps found after the last release. (#4525)
- The Codex readiness check and the second-model review now pick the same model for a custom Codex provider (GST-3089). (#4491)
- gstack's Codex hooks work when your Codex folder's path has a `(`, `&` or other shell character in it. (#4528)
- Uninstalling gstack keeps your own Codex hooks trusted. (#4528)
- Codex hook setup matches Codex more closely. (#4528)
- On Windows, setup copies the Claude Code hook code that gstack's Codex hooks import (#4528)
- /review's checklist on Codex reads shell commands more carefully. (#4528)
- On Codex, gstack now suggests the right skill for a prompt and stops destructive commands before they run. (#4512)
- On Codex, /review is now held once when it tries to finish without its adversarial pass or its report, as it already is on Claude Code. (#4522)
- On Codex, gstack's skill suggestion now names the skill the way Codex lists it. (#4523)
- On Codex, gstack's suggested prompts and docs now name skills the way Codex lists them. (#4524)
- /review on Codex with gpt-6.1-sol is no longer held back for a second-model pass it already ran. (#4526)
- On Codex, office-hours, plan-design-review and gstack-upgrade work again when you name them. (#4510)
- gstack knows when Codex is running unattended. (#4510)
- Codex reviewers start with a clean context. (#4510)
- Docs no longer say Codex has no hook support. (#4510)
- A command whose output was cut off no longer counts as a finished read. (#4492)
- A folder marked as published data can no longer hide source code or agent files from /ship's review. (#4516)
- Correction to the on-by-default helper hooks entry: the hook still routes only to /review, /qa and /verify, so skills you've installed yourself aren't suggested yet. With gstack's skills it routes 14 of 60 real bug reports (was 17), and long prompts are no longer skipped. (#4505)
- /sync-gbrain's read check now works when the first page in your code source is over 64 KB. (#4497)
- Free-suite CI no longer fails before any test runs when Ubuntu's package mirror is slow. (#4530)
- Syncing a second repo no longer makes the first one ask for /sync-gbrain again. (#4532)
- gstack's helper hooks are now on by default in Claude Code, OpenClaw and Hermes, and the prompt-routing hook fires on real bug reports. (#4503)
- `gstack-pregate clear --receipt` accepts a remote gate run from before a merged release when the release only regenerated a declared release output. (#4483)
- The second-model review now keeps the reviewer's own result, and reports a used-up budget as a budget stop. (#4515)
- When the reviewer wrote its review to the result file it was asked for, Codex then replaced that file with its short closing message ("Done, I wrote the review"), so the review was thrown away and reported as missing its markers. (#4515)
- When the provider refuses the call with HTTP 402 Payment Required (usually a metering proxy that hit its allowance, or an account out of credits or quota), gstack used to record only "exit 1". (#4515)
- The pre-commit /verify check now fails a check that times out, and its checks share one time budget (GST-3201). (#4527)
- /ship's review collector no longer reads a reviewer's own scope list or list of passed checks as findings (GST-3201). (#4527)
- Merge-queue stack tests fail when git cannot list a PR's added test files (GST-3201). (#4527)
- /ship can now review results PRs that publish raw session data. (#4511)
- /ship's review pass works when the checkout you ship isn't the project's default checkout. (#4507)
- A second review round no longer reports done when nothing was reviewed. (#4520)
- The prompt-routing hook now suggests skills you install yourself, and no longer sends ordinary bug reports to /qa (GST-3123). (#4506)
- gstack's script runner and merge-queue checks now also refuse cut-off output. (#4502)
- A failed second-model review reports the cause that actually stopped it, and the runner never deletes its own prompt. (#4519)
- `gstack-ship-queue ready` now fills in the trial merges and the second-model trail read instead of printing lines to fill. (#4529)
- Five small ship-tool fixes. (#4529)
- A ruling the owner gives while a `/ship` run is open can be relayed. (#4529)
- /ship no longer kills a working test run as stalled when the test command starts its workers in their own session. (#4533)
- /ship no longer stops a review because the state-machine reader started its reply the way its instructions say. (#4504)
- When Claude Code runs /verify right before a commit, it now gets a quick check instead of the full fix loop (GST-3150). (#4508)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant