Skip to content

Deep Scan: a --max-cost stop after discovery is sealed as scan.status: completed (SARIF executionSuccessful: true, exit 0) #2415

Open

Description

@MatteoManzoni

Summary

When a Deep Scan crosses --max-cost (limits.max_cost_usd_per_scan, action input max-cost) after discovery has finished, the remaining candidates are not validated, but the bundle is sealed with scan.status: "completed". The scan contract defines completed as "The requested scan reached normal completion" and says "Only completed supports a completed-scan conclusion". The SARIF export reports executionSuccessful: true, the CLI emits scan.completed and exits 0.

If the same cost stop lands while discovery is still running, the scan ends as scan.failed classification="cost_limit_exceeded" with exit 2 and retained artifacts. So one condition, the estimated cost crossing the limit, is reported either as a failure or as a normal completion, depending only on which phase was running at that moment.

Current behaviour

Checked on npm-v0.8.3 and main at 9b3ce7a. The first case was observed in CI on 0.8.2, and the code path is unchanged since then.

Cost stop during discovery

  • complete-budget-exhausted-scan refuses ("Budget-exhausted scan completion requires successfully completed Deep Scan discovery."). The SDK falls through and the scan is stopped as failed.
  • The CLI emits scan.failed classification="cost_limit_exceeded" max_cost_usd=… estimated_usd=… and exits 2.
  • This matches the contract: a stopped outcome, with absence of findings treated as inconclusive.

Cost stop after discovery finished (terminal_reason is saturated or capped)

  • sdk/typescript/src/api.ts catches ScanCostLimitExceededError and runs workbench_db.py complete-budget-exhausted-scan.
  • budget_exhausted_draft (plugins/codex-security/scripts/workbench_saved_results.py) does two things:
    • It records unvalidated candidates as needs_follow_up surfaces, with deferred reasons starting "Validation was deferred because the scan reached its cost limit".
    • It adds a generic deferred entry with id scan-cost-limit.
  • complete_scan_locked then seals the bundle with the default status completed.
  • finalize_scan_contract.py sets execution_successful = manifest["scan"]["status"] == "completed", so SARIF reports executionSuccessful: true. The cost warning appears only as a toolExecutionNotifications message. test_budget_exhaustion_preserves_validated_standard_results_without_candidate_ledgers asserts this. test_budget_exhaustion_with_no_candidates_is_honestly_partial seals a cost-stopped scan with zero findings as complete; the only marker is the scan-cost-limit deferred entry.
  • The CLI emits scan.completed, which has no stop-reason field, and exits 0 (or 1 when findings meet a configured failure threshold), as sdk/typescript/docs/cli.md documents. readDeepScanStop knows the reason ("Reached the … cost limit. More issues may remain."), but it is printed only in the human summary.

Why it matters

  • Automation keyed on scan.status, SARIF executionSuccessful, the exit code, or the scan.completed diagnostic treats a cost-stopped scan as complete. Today, telling the two apart means string-matching warning or deferred-reason text, or relying on the scan-cost-limit id, which is not documented.
  • Candidates that were discovered but not validated are not SARIF results. Suppose a SARIF consumer closes alerts that are missing from a newer successful analysis. It can then close alerts this run never re-checked.
  • It contradicts the contract's own rule for stopped outcomes: consumers should treat absence of findings as inconclusive.

Proposed change (minimal)

  1. Do not seal a cost-limit stop as completed. Keep the sealed report and the deferred candidates exactly as today, but use a stopped status. interrupted ("stopped before normal completion … retained artifacts may be partial") already fits, or there could be a dedicated status. The CLI already has a branch for a sealed non-completed result: it prints "Scan ; available results were retained." and exits 2.
  2. If completed has to stay for compatibility, add a documented machine-readable stop reason to the sealed manifest and the JSON result, for example scan.stopReason: "cost_limit". Also report executionSuccessful: false for it in SARIF, and include the stop reason in the scan.completed / scan.failed diagnostic.
  3. Describe both cost-stop outcomes (during discovery, after discovery) in one place: the scan.status table in plugins/codex-security/references/scan-contract.md and the --max-cost section of sdk/typescript/docs/cli.md.

Either 1 or 2 would let consumers tell "stopped at the cost limit" from "finished" without parsing text.

Activity

  1. mldangelo-oai commented on Nov 15, 2026

    @mldangelo-oai
    Collaborator

    I reproduced this on both a fresh scan and a resumed one. On main, hitting the cost limit after discovery produced a completed manifest, successful SARIF, and exit 0. #2492 corrects all three and exits 2.

    There’s one recovery case to fix before closing this. When a database write failed after the report was sealed, SARIF kept the synthetic finding but the workbench showed zero findings. Retrying should bring the workbench back in sync with the sealed report. I’m keeping this open with #2492.

  2. added
    bugSomething isn't working
    priority:p1High-impact scan failure, integrity gap, or user-blocking regression
    area:costToken usage, cost estimates, spending limits, and quota visibility.
    area:deep-scanDeep Scan coordination, worker scheduling, coverage, convergence, and resume.
    on Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:costToken usage, cost estimates, spending limits, and quota visibility.area:deep-scanDeep Scan coordination, worker scheduling, coverage, convergence, and resume.bugSomething isn't workingpriority:p1High-impact scan failure, integrity gap, or user-blocking regression

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions