Repository navigation
FIX: Treat provider feature rejections like OpenRouter 404s as unsupported in capability detection - #4490
Open
Richard Lundeen (richlundeen) wants to merge 1 commit into
Conversation
… capability detection After strict discovery's control request passes, a probe that fails with HTTP 400, 404, 415, 422, or 501 (read from the error or its cause chain) marks that feature or input unsupported and is not retried. Auth, throttling, timeouts, other 5xx, and network errors still abort detection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
There are no active review issues blocking approval of the capability-discovery update.
No open issues 🎉
What changed in this PR
Updates PyRIT’s shared capability discovery so the GUI treats provider feature rejections, including OpenRouter 404s, as unsupported capabilities after a successful control request. Operational failures remain inconclusive rather than changing capabilities.
Changes:
- Adds HTTP 404, 415, and 501 to feature-rejection handling and checks exception cause chains for status codes.
- Stops discovery-level retries for recognized request rejections.
- Expands regression coverage and documents strict-discovery behavior.
| File | Description |
|---|---|
| tests/unit/prompt_target/test_discover_target_capabilities.py | Covers additional statuses, wrapped errors, capability preservation, and rejection retry behavior. |
| pyrit/prompt_target/common/discover_target_capabilities.py | Centralizes rejection classification and skips retries for rejected requests. |
| doc/code/targets/0_prompt_targets.md | Explains provider rejection statuses and operational-failure handling. |
Review effort: Balanced
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Strict capability detection (the GUI's "detect capabilities" button) could only treat a failed feature probe as "unsupported" when the provider answered with HTTP 400 or 422. OpenRouter answers unsupported image or audio input with 404 (
No endpoints found that support image input), so detection aborted as inconclusive for those targets. This change classifies provider rejections more generally while keeping operational failures inconclusive.__cause__chain, so rawopenai/httpxerrors and PyRIT or LiteLLM wrappers such asPyritException(status_code=404)are classified the same way. AValueErrorwithout a status still counts as PyRIT's own request validation.Related: #4486 edits
_raise_if_inconclusiveand its error messages in the same file. This PR keeps its diff small so the two can merge in either order.