Skip to content

v1.93.41.0 Codex outside reviews report their token usage (#2948) - #3729

Merged
capy-ai[bot] merged 4 commits into
mainfrom
gsta214/d
Nov 3, 2026
Merged

capy-ai[bot] merged 4 commits into
mainfrom
gsta214/d

Conversation

@garrytan

@garrytan garrytan commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner
{
  "schema_version": 1,
  "pr": 3729,
  "head": "b711c566f70dc58188cbb415284d5c555ca3b168",
  "version": "1.91.378.0",
  "base": "main",
  "gate": {
    "typecheck": {
      "pass": 1,
      "fail": 0,
      "skip": 0
    },
    "focused": {
      "pass": 130,
      "fail": 0,
      "skip": 0
    }
  },
  "session_kind": "unattended",
  "head_tree": "96ee40a924edb45930d1cf04d945cfcfca1ab703",
  "product_patch_id": "dffb05e540177f4d73190617c14926316a67128b",
  "gstack_version": "1.93.22.0",
  "review_depth": {
    "depth": "full",
    "class": "code",
    "runtime_lines": 63,
    "rules": "b2272efd321ce7cb",
    "base": "82db89ab3ea3c64185538b8af33da2ffd1a18b57",
    "registry": "82db89ab3ea3c64185538b8af33da2ffd1a18b57",
    "gate_scan": {
      "tagged": 0,
      "head": "b711c566f70dc58188cbb415284d5c555ca3b168"
    },
    "reasons": [
      "risk:money",
      "runtime=63"
    ],
    "risk": [
      "money"
    ]
  },
  "pregate": "requires-remote cleared @96ee40a924ed (1 checks, current for b711c566f70d)",
  "requires_remote": [
    "free-tests.yml/macos-named-regressions: https://github.com/garrytan/gstack/actions/runs/41235153282",
    "windows-free-tests.yml/windows-free-shard: https://github.com/garrytan/gstack/actions/runs/41235153176",
    "pregate.json/remote_gate: remote-gate tree=96ee40a924ed tests=5370"
  ],
  "issues": {
    "fixes": [],
    "refs": [
      2948
    ],
    "flakes": []
  },
  "docs": {
    "status": "updated",
    "scope": "docs/ and README.md (131 docs), base 3bfe566 to 234f6be"
  },
  "review_loop": {
    "passes": 1,
    "blocking_found": 2,
    "deferred": 4,
    "noted": 0,
    "fixed": 0,
    "scope": "frozen",
    "rounds": 1,
    "unrecorded": 2,
    "ledgers_read": 1
  },
  "review_targeted": {
    "classes": [
      "money"
    ],
    "rules": "c82addff6fb60fa0",
    "brief_sha": "7f92a0d0a8319c3c0e076fb3b1a4689c45120b2be2c14dfc7d4cb03aafa9d6a5",
    "product_patch_id": "dffb05e540177f4d73190617c14926316a67128b",
    "status": "done",
    "findings": 0,
    "raised_by": "targeted"
  }
}

What changed

Codex outside reviews now print the token usage Codex reported, on one more stdout line after the verdict.

Before/after as text: Codex outside reviews now print the token usage Codex reported (OUTSIDE_USAGE), which gstack used to discard
--- Before · before.txt · captured output
+++ After · after.txt · captured output
 $ bun lib/outside-review-result.ts --events events.jsonl review text
 VERDICT: clean
 FINDINGS: none
+OUTSIDE_USAGE: input=24763 cached_input=24448 output=122 reasoning_output=64
 

(The before/after image could not be rendered on this machine, so the text version stands in for it.)

Why

#2948 asked for Codex review token counts: Codex already reports them in its --json events, and gstack threw them away. Now every Codex outside voice prints OUTSIDE_USAGE: with the counters Codex sent, so a reviewer's cost shows up next to its verdict. There is one usage parser instead of two, and it no longer undercounts, because it takes the cumulative snapshot, or the latest turn.completed total, which Codex already reports as the running session total.

Blast radius

The CLI change reaches every skill that runs lib/outside-review-result.ts with --events: /review, /ship, /autoplan, the plan reviews, /office-hours, /spec, /design-review, /document-release and the /codex review, challenge and consult modes. Each one prints one more stdout line. That is safe because every reader of this output matches single lines by prefix (for example test/helpers/outside-voice-evidence.ts:208), except one test that pinned the generated plan-review invocation's whole stdout (test/run-in-background-guidance.test.ts), which CI caught and this PR updates. The verdict, findings and exit code do not change. Checked by running the CLI on every verdict, with a missing events file and without --events, plus the focused tests. lib/outside-voice-runner.ts now imports the parser and re-exports it, so there is no import cycle. RunnerUsage only gains optional fields.

Behavior changes

  • Every Codex outside voice run with --events prints one more stdout line, OUTSIDE_USAGE: input=… cached_input=… output=… reasoning_output=… (only the counters Codex sent), or OUTSIDE_USAGE: unknown.
  • Runs without --events print exactly what they printed before.

Approvals

Date Asked Exact words Outcome Covers Still pending
2026-10-31 Approve the plan "Small feature requests from the 2026-10-31 cleanup (GST-531)" (autoplan run autoplan-20261031175328, 11 decisions; revised plan carried at gate_rev 2) “Do it” (Garry Tan, relayed by agent:jam_01M69H8PKK07YCHT6WCQ9G87ZK) approved the reviewed plan, every recommendation taken none

Summary

Codex reports token counts in its JSON events, and #2948 asked gstack to stop discarding them. The parser moved into lib/outside-review-result.ts beside the verdict code, keeps every counter Codex sends (cached, cache-write, reasoning and total too), and the CLI prints them as OUTSIDE_USAGE: whenever it is given an events file. docs/troubleshooting.md names the new line.

Test Coverage

CODE PATHS                                              TESTS
[+] usageFromEvents(events)
  ├── turn.completed usage, single turn                 [★★ TESTED] outside-review-result.test.ts "one turn"
  ├── malformed / unrelated lines skipped, latest wins  [★★★ TESTED] "the latest turn wins"
  ├── cumulative snapshot via msg.info wins             [★★ TESTED] "a cumulative total_token_usage snapshot"
  ├── snapshot via info / payload.info roots            [GAP] only msg.info is exercised
  └── no usage at all -> counters absent                [★★★ TESTED] CLI "OUTSIDE_USAGE: unknown"
[+] formatUsageLine(usage)                              [★★ TESTED] CLI cases
[~] CLI prints the line only with --events              [★★ TESTED] CLI case without --events
[~] outside-voice-runner codex-cli usage propagation    [★★ TESTED] outside-runner-contract.test.ts:146-157

Tests: 1571 → 1571 test files (the four new tests extend test/outside-review-result.test.ts).
Coverage: 88% value-weighted (7 of 8 paths).
Test value: 0 tests written by the audit, 0 rejected, 0 existing tests extended, 0 paths weakly covered.
Proposed test (not written, below the blocking floor): extend the snapshot test with info.total_token_usage and payload.info.total_token_usage rows. Value: protects=all three snapshot roots; fails_when=a root is dropped from the ?? chain; seam=none.
Tests ran: typecheck count unavailable
Tests ran: focused 130

Pre-Landing Review

Re-shipped on 2026-11-02 at full depth (risk: money, from the updated review-depth rules), one round at b711c56. No issues found at the blocking floor. Specialists (testing, security, state-machines), Red Team and the targeted money reader: no findings. The cross-family outside voice (openai/gpt-6.1-sol as a Capy subagent; the Codex CLI is not installed here): no findings, "approve". The native adversarial pass found five Low, non-blocking telemetry notes, listed below.
REVIEW_LOOP: passes=1 blocking_found=2 deferred=4 noted=0 fixed=0 scope=frozen unrecorded=2 ledgers_read=1
(The two "unrecorded" items are adversarial notes #3 and #4 below. The reader labeled them FIXABLE, so the ledger collects them as CRITICAL, but they name no file:line, so no review-log row can pair with them. Both are deferred here and marked "Blocks: no".)
Gate edits: none detected (gate scan tagged 0 hunks at b711c56)

Earlier fix (2026-10-31): CI free-suite shard 18 failed 8 cases in test/run-in-background-guidance.test.ts, which pins the generated Codex plan-review invocation's exact stdout. Its expected output now includes OUTSIDE_USAGE: unknown, and no product code changed. A fix-check reviewer read that fix and found nothing.

Deferred findings

All five are telemetry-only. None changes a verdict, an exit code or a gate.

  1. Killed or timed-out Codex runs have no turn.completed event, so they print OUTSIDE_USAGE: unknown even though tokens were spent. The follow-up that stores usage must read unknown as "not measured", never as 0.
  2. lib/outside-review-result.ts:316: in a file that mixes both formats, an early total_token_usage snapshot outranks a later turn.completed total. codex exec --json does not emit snapshots today.
  3. input= already includes cached_input= (OpenAI semantics), and the line does not say so. Adding the two double-counts.
  4. An unreadable --events file prints unknown, the same as a run that reported nothing.
  5. The latest-turn rule assumes Codex reports the running session total in turn.completed. This was not checked against a live codex exec. If the assumption is wrong, the count comes out low, not corrupted.

Exploratory QA

Smoke (CLI, report-only): with a Codex turn.completed events file the CLI prints OUTSIDE_USAGE: input=24763 cached_input=24448 output=122 reasoning_output=64 on a clean verdict, a findings verdict and an unavailable verdict; a missing events file prints OUTSIDE_USAGE: unknown; without --events no line is printed; exit codes are unchanged in every case. Not run: a live codex exec (the Codex CLI is not installed on this machine).

Design Review

No frontend files changed — design review skipped.

Eval Results

No prompt-related files changed — evals skipped.

Plan Completion

Plan completion audit: not run (no plan is bound to this branch and no docs/designs/ file matches). The approved plan is the GST-531 autoplan run named under Approvals.

Issues

Refs #2948

Storing usage in review-log rows, the native codex review rollout path and a token column in /ship are follow-ups, so this PR does not close #2948.

TODOS

No TODO items completed in this PR.

Documentation

Status: updated. One factual doc edit; no blockers.

Scope: committed changes base 3bfe566 to HEAD 234f6be (lib/outside-review-result.ts, lib/outside-voice-runner.ts, changelog fragment, test). Nothing staged, unstaged or untracked.

Documentation health:
docs/troubleshooting.md [Updated] added the OUTSIDE_USAGE stdout line to the outside-review section
changelog.d/gsta214-d.md [Current] not edited (out of audit scope)

Documentation coverage: OUTSIDE_USAGE has reference coverage in docs/troubleshooting.md only; no how-to or tutorial gap is claimed. Diagram drift: none checked against this diff.

Test plan

  • bun run typecheck: pass
  • bun test test/outside-review-result.test.ts test/outside-runner-contract.test.ts test/outside-voice-invocation.test.ts test/run-in-background-guidance.test.ts: 130 pass, 0 fail
  • Remote pregate (Ubicloud, UBI_OWNER=gsta214) on tree 96ee40a924ed: 5370 tests, 0 failed, $0.05
  • macOS named regressions (run 41235153282) and the Windows free shards (run 41235153176): cleared from this PR's CI at b711c56

garrytan and others added 2 commits October 31, 2026 18:32
…ne (#2948)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

…#2948)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@github-actions github-actions Bot changed the title Codex outside reviews report their token usage (#2948) v1.93.41.0 Codex outside reviews report their token usage (#2948) Nov 3, 2026
@capy-ai
capy-ai Bot merged commit 106fd39 into main Nov 3, 2026
45 checks passed
@omicron-soft

Copy link
Copy Markdown

Amazing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Codex reviews record no token usage, though Codex persists it on disk for every run

2 participants