Repository navigation
v1.93.41.0 Codex outside reviews report their token usage (#2948) - #3729
Merged
Merged
Conversation
…ne (#2948) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
|
😎 This pull request was merged. |
…#2948) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
5 tasks done
|
Amazing! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Codex outside reviews now print the token usage Codex reported, on one more stdout line after the verdict.
Before/after as text: Codex outside reviews now print the token usage Codex reported (OUTSIDE_USAGE), which gstack used to discard
(The before/after image could not be rendered on this machine, so the text version stands in for it.)
Why
#2948 asked for Codex review token counts: Codex already reports them in its
--jsonevents, and gstack threw them away. Now every Codex outside voice printsOUTSIDE_USAGE:with the counters Codex sent, so a reviewer's cost shows up next to its verdict. There is one usage parser instead of two, and it no longer undercounts, because it takes the cumulative snapshot, or the latestturn.completedtotal, which Codex already reports as the running session total.Blast radius
The CLI change reaches every skill that runs
lib/outside-review-result.tswith--events: /review, /ship, /autoplan, the plan reviews, /office-hours, /spec, /design-review, /document-release and the /codex review, challenge and consult modes. Each one prints one more stdout line. That is safe because every reader of this output matches single lines by prefix (for exampletest/helpers/outside-voice-evidence.ts:208), except one test that pinned the generated plan-review invocation's whole stdout (test/run-in-background-guidance.test.ts), which CI caught and this PR updates. The verdict, findings and exit code do not change. Checked by running the CLI on every verdict, with a missing events file and without--events, plus the focused tests.lib/outside-voice-runner.tsnow imports the parser and re-exports it, so there is no import cycle.RunnerUsageonly gains optional fields.Behavior changes
--eventsprints one more stdout line,OUTSIDE_USAGE: input=… cached_input=… output=… reasoning_output=…(only the counters Codex sent), orOUTSIDE_USAGE: unknown.--eventsprint exactly what they printed before.Approvals
Summary
Codex reports token counts in its JSON events, and #2948 asked gstack to stop discarding them. The parser moved into
lib/outside-review-result.tsbeside the verdict code, keeps every counter Codex sends (cached, cache-write, reasoning and total too), and the CLI prints them asOUTSIDE_USAGE:whenever it is given an events file. docs/troubleshooting.md names the new line.Test Coverage
Tests: 1571 → 1571 test files (the four new tests extend test/outside-review-result.test.ts).
Coverage: 88% value-weighted (7 of 8 paths).
Test value: 0 tests written by the audit, 0 rejected, 0 existing tests extended, 0 paths weakly covered.
Proposed test (not written, below the blocking floor): extend the snapshot test with
info.total_token_usageandpayload.info.total_token_usagerows. Value: protects=all three snapshot roots; fails_when=a root is dropped from the??chain; seam=none.Tests ran: typecheck count unavailable
Tests ran: focused 130
Pre-Landing Review
Re-shipped on 2026-11-02 at full depth (risk: money, from the updated review-depth rules), one round at b711c56. No issues found at the blocking floor. Specialists (testing, security, state-machines), Red Team and the targeted money reader: no findings. The cross-family outside voice (openai/gpt-6.1-sol as a Capy subagent; the Codex CLI is not installed here): no findings, "approve". The native adversarial pass found five Low, non-blocking telemetry notes, listed below.
REVIEW_LOOP: passes=1 blocking_found=2 deferred=4 noted=0 fixed=0 scope=frozen unrecorded=2 ledgers_read=1
(The two "unrecorded" items are adversarial notes #3 and #4 below. The reader labeled them FIXABLE, so the ledger collects them as CRITICAL, but they name no file:line, so no review-log row can pair with them. Both are deferred here and marked "Blocks: no".)
Gate edits: none detected (gate scan tagged 0 hunks at b711c56)
Earlier fix (2026-10-31): CI free-suite shard 18 failed 8 cases in
test/run-in-background-guidance.test.ts, which pins the generated Codex plan-review invocation's exact stdout. Its expected output now includesOUTSIDE_USAGE: unknown, and no product code changed. A fix-check reviewer read that fix and found nothing.Deferred findings
All five are telemetry-only. None changes a verdict, an exit code or a gate.
turn.completedevent, so they printOUTSIDE_USAGE: unknowneven though tokens were spent. The follow-up that stores usage must readunknownas "not measured", never as 0.lib/outside-review-result.ts:316: in a file that mixes both formats, an earlytotal_token_usagesnapshot outranks a laterturn.completedtotal.codex exec --jsondoes not emit snapshots today.input=already includescached_input=(OpenAI semantics), and the line does not say so. Adding the two double-counts.--eventsfile printsunknown, the same as a run that reported nothing.turn.completed. This was not checked against a livecodex exec. If the assumption is wrong, the count comes out low, not corrupted.Exploratory QA
Smoke (CLI, report-only): with a Codex
turn.completedevents file the CLI printsOUTSIDE_USAGE: input=24763 cached_input=24448 output=122 reasoning_output=64on a clean verdict, a findings verdict and an unavailable verdict; a missing events file printsOUTSIDE_USAGE: unknown; without--eventsno line is printed; exit codes are unchanged in every case. Not run: a livecodex exec(the Codex CLI is not installed on this machine).Design Review
No frontend files changed — design review skipped.
Eval Results
No prompt-related files changed — evals skipped.
Plan Completion
Plan completion audit: not run (no plan is bound to this branch and no docs/designs/ file matches). The approved plan is the GST-531 autoplan run named under Approvals.
Issues
Refs #2948
Storing usage in review-log rows, the native
codex reviewrollout path and a token column in /ship are follow-ups, so this PR does not close #2948.TODOS
No TODO items completed in this PR.
Documentation
Status: updated. One factual doc edit; no blockers.
Scope: committed changes base 3bfe566 to HEAD 234f6be (lib/outside-review-result.ts, lib/outside-voice-runner.ts, changelog fragment, test). Nothing staged, unstaged or untracked.
Documentation health:
docs/troubleshooting.md [Updated] added the OUTSIDE_USAGE stdout line to the outside-review section
changelog.d/gsta214-d.md [Current] not edited (out of audit scope)
Documentation coverage: OUTSIDE_USAGE has reference coverage in docs/troubleshooting.md only; no how-to or tutorial gap is claimed. Diagram drift: none checked against this diff.
Test plan
bun run typecheck: passbun test test/outside-review-result.test.ts test/outside-runner-contract.test.ts test/outside-voice-invocation.test.ts test/run-in-background-guidance.test.ts: 130 pass, 0 failUBI_OWNER=gsta214) on tree 96ee40a924ed: 5370 tests, 0 failed, $0.05