Skip to content

FIX: Treat provider feature rejections like OpenRouter 404s as unsupported in capability detection - #4490

Open
Richard Lundeen (richlundeen) wants to merge 1 commit into
microsoft:mainfrom
richlundeen:richlundeen-detect-feature-rejection
Open

Richard Lundeen (richlundeen) wants to merge 1 commit into
microsoft:mainfrom
richlundeen:richlundeen-detect-feature-rejection

Conversation

@richlundeen

Copy link
Copy Markdown
Contributor

Description

Strict capability detection (the GUI's "detect capabilities" button) could only treat a failed feature probe as "unsupported" when the provider answered with HTTP 400 or 422. OpenRouter answers unsupported image or audio input with 404 (No endpoints found that support image input), so detection aborted as inconclusive for those targets. This change classifies provider rejections more generally while keeping operational failures inconclusive.

  • Strict discovery already sends a control request to the same endpoint, model, and credentials before any feature probe. After that passes, a probe failing with HTTP 400, 404, 415, 422, or 501 now means the provider refused that feature or input. Providers use different codes for this: OpenAI and Azure send 400, OpenRouter sends 404, validation layers send 422, media checks send 415, and some servers send 501 for features they do not implement.
  • Authentication and permission (401, 403, 407), timeouts (408), conflicts (409), throttling (429), other 5xx statuses, and network errors still abort detection without changing capabilities. A control request that fails for any reason still aborts.
  • The status is read from the error or from its __cause__ chain, so raw openai/httpx errors and PyRIT or LiteLLM wrappers such as PyritException(status_code=404) are classified the same way. A ValueError without a status still counts as PyRIT's own request validation.
  • Capability flag probes, modality probes, the tool-call history probe, and the control request all go through the same check. A rejected request is no longer retried, because resending it cannot change the answer. Best-effort mode behaves as before.
  • Updated the strict-discovery note in the prompt targets doc.

Related: #4486 edits _raise_if_inconclusive and its error messages in the same file. This PR keeps its diff small so the two can merge in either order.

… capability detection

After strict discovery's control request passes, a probe that fails with
HTTP 400, 404, 415, 422, or 501 (read from the error or its cause chain)
marks that feature or input unsupported and is not retried. Auth, throttling,
timeouts, other 5xx, and network errors still abort detection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 21, 2026 05:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues blocking approval of the capability-discovery update.

No open issues 🎉

What changed in this PR

Updates PyRIT’s shared capability discovery so the GUI treats provider feature rejections, including OpenRouter 404s, as unsupported capabilities after a successful control request. Operational failures remain inconclusive rather than changing capabilities.

Changes:

  • Adds HTTP 404, 415, and 501 to feature-rejection handling and checks exception cause chains for status codes.
  • Stops discovery-level retries for recognized request rejections.
  • Expands regression coverage and documents strict-discovery behavior.
File Description
tests/​unit/​prompt_target/​test_discover_target_capabilities.py Covers additional statuses, wrapped errors, capability preservation, and rejection retry behavior.
pyrit/​prompt_target/​common/​discover_target_capabilities.py Centralizes rejection classification and skips retries for rejected requests.
doc/​code/​targets/​0_prompt_targets.md Explains provider rejection statuses and operational-failure handling.

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants