Skip to content

fix(deep-scan): report cost-limit exhaustion as stopped - #2492

Closed
Aditya-linux wants to merge 3 commits into
openai:mainfrom
Aditya-linux:fix/deep-scan-cost-limit-status
Closed

Aditya-linux wants to merge 3 commits into
openai:mainfrom
Aditya-linux:fix/deep-scan-cost-limit-status

Conversation

@Aditya-linux

@Aditya-linux Aditya-linux commented Nov 9, 2026 •

Copy link
Copy Markdown

Summary

When Deep Scan finishes discovery and hits its cost limit during validation, a completed result can make an empty report look conclusive. This change records the cost stop as a failed scan and retains the results available at that point. Addresses #2415.

Changes

The workbench seals the parent draft with the measured cost, its warning, validated findings and deferred candidates. SARIF records executionSuccessful: false. The contributor incorporated the checkpoint-preservation follow-up in afd673885, which keeps a later preservation attempt from replacing an already sealed checkpoint.

Testing

On the published head 6b1dc16ee, four focused workbench cases passed for authored findings, unvalidated discovery and empty results. Two additional cases reproduced the recovery issues below. The contributor also reported 34 focused Python cases passing after merging main. Contributor’s test results

The plugin source-contract job failed its file-size check when CI tested this branch with a newer base. CI job

Risk and rollout

Two cases still need fixes before rollout. If SQLite fails after the manifest is sealed but before it records the failed status, the workbench still shows a running scan and retrying the cost stop fails. A parent draft containing a validated finding with complete: false also fails to finalize. Consumers should treat an empty stopped scan as inconclusive.

Public disclosure review

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

The current review covered the branch, commits, changes, discussion and linked check results. A full review of historical CI logs and attachments remains open.

@github-actions github-actions Bot added the bug Something isn't working label Nov 9, 2026

@mangeshraut712 mangeshraut712 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reporting this as a stopped outcome (exit 2, executionSuccessful: false) matches the scan contract's rule that only completed supports a completed-scan conclusion. Two things before this is ready, going by the Testing note.

  1. The contract and CLI doc changes here say the stopped bundle preserves validated findings and deferred candidates. But by your own description that only holds with the follow-up commit, and on this head 8/41 Python and 3 SDK cases fail. Could you push the follow-up onto this branch so the docs and the code land together? Otherwise the contract would describe behaviour the merged code doesn't have.

  2. See the inline note on the tokenless failure call. It may explain part of those failures.

Copy link
Copy Markdown
Author

Thanks for the review. I cherry-picked the supplied checkpoint-preservation follow-up onto this branch as afd673885. The branch now preserves the authoritative findings/deferred checkpoint and removes the fail_scan_locked call discussed inline. I also updated the PR description to reflect the combined implementation and validation.

…mit-status

# Conflicts:
#	plugins/codex-security/scripts/workbench_db.py

Copy link
Copy Markdown
Author

Also merged current main in 6b1dc16ee and resolved the workbench_db.py conflict by retaining both the new upstream log_completed_at field and the cost-stop status/failure fields. Post-merge validation: 34 focused budget-exhaustion Python cases passed (the Unicode case under UTF-8 mode), Ruff and TypeScript compile passed, and the plugin source compatibility suite passed (8 passed, 1 platform skip). The PR is mergeable with no unresolved review threads; fresh CI is awaiting GitHub workflow scheduling/approval.

Copy link
Copy Markdown
Collaborator

Continued this work in #2686, preserving the commits from this branch. The follow-up reconciles the workbench index with the sealed findings after an interrupted SQLite write and retains validated findings from explicitly incomplete drafts. The fresh and resumed CLI regressions and the portable plugin checks passed.

@Aditya-linux

Copy link
Copy Markdown
Author

Thanks for continuing this in #2686 and preserving my commits. I'm closing #2492 to avoid duplicate work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants