Repository navigation
Description
Describe the bug
When a target's content filter blocks a request, PyRIT targets don't raise. handle_bad_request_exception (pyrit/exceptions/exception_classes.py:507) returns a normal response whose piece has converted_value_data_type="error", response_error="blocked", and the provider's error JSON as its value. Two converter paths don't check for this:
1. LLM-backed converters return the error JSON as the "converted" prompt.
LLMGenericTextConverter._send_with_retries_async (pyrit/converter/llm_generic_text_converter.py:312 and :334) does self._process_response(response[0].get_value()) without looking at response_error or the data type.
- So when the converter's own LLM is content-filtered,
ToneConverter,TranslationConverter,TenseConverter,NoiseConverter,VariationConverter(and the generic converter) return the error JSON withoutput_type="text". - The attack then sends that JSON to the objective target as the attack prompt.
- Because nothing raises, the converter's retry logic never runs either.
- This is easy to hit in practice, since the converter LLM is often a filtered Azure OpenAI deployment.
2. Response converters crash the attack when the objective target's reply is blocked.
PromptNormalizer.send_prompt_async (pyrit/prompt_normalizer/prompt_normalizer.py:267) runs the response converters on the last response.
ConverterConfigurationdefaults toprompt_data_types_to_apply=None, i.e. every piece type, so the converters receive theerrorpiece.- Text-only converters (ROT13, Base64, and
TranslationConverter, which the converters docs use as the round-trip response converter) raiseValueError: Input type error not supported. - The attack fails. The user turn is already in memory, but the blocked reply is never saved, so the conversation is left with an orphaned user turn.
- Without response converters, the same blocked reply is saved and the attack finishes normally.
Workaround for part 2: pass prompt_data_types_to_apply=["text"] on the response ConverterConfiguration.
Steps/Code to Reproduce
Offline; the target imitates a content-filtered OpenAI target using PyRIT's own handle_bad_request_exception.
import asyncio
from pyrit.converter import ROT13Converter, ToneConverter
from pyrit.exceptions import handle_bad_request_exception
from pyrit.executor.attack import AttackConverterConfig, PromptSendingAttack
from pyrit.memory import CentralMemory
from pyrit.prompt_normalizer import ConverterConfiguration
from pyrit.prompt_target import PromptTarget
from pyrit.prompt_target.common.target_capabilities import TargetCapabilities
from pyrit.prompt_target.common.target_configuration import TargetConfiguration
from pyrit.setup import IN_MEMORY, initialize_pyrit_async
class BlockingTarget(PromptTarget):
"""Behaves like an OpenAI target whose content filter blocks every request."""
_DEFAULT_CONFIGURATION = TargetConfiguration(
capabilities=TargetCapabilities(
supports_multi_turn=True,
supports_multi_message_pieces=True,
supports_system_prompt=True,
supports_editable_history=True,
)
)
async def _send_prompt_to_target_async(self, *, normalized_conversation):
request = normalized_conversation[-1].message_pieces[0]
return [
handle_bad_request_exception(
response_text='{"error": {"code": "content_filter", "message": "The response was filtered"}}',
request=request,
error_code=400,
is_content_filter=True,
)
]
async def main():
await initialize_pyrit_async(memory_db_type=IN_MEMORY, silent=True)
# Part 1: the converter's own LLM is content-filtered.
result = await ToneConverter(converter_target=BlockingTarget(), tone="angry").convert_async(prompt="hello")
print("Part 1 ToneConverter ->", result.output_type, result.output_text)
# Part 2: the objective target is content-filtered and a response converter is configured.
for response_converters in ([], [ConverterConfiguration(converters=[ROT13Converter()])]):
attack = PromptSendingAttack(
objective_target=BlockingTarget(),
attack_converter_config=AttackConverterConfig(response_converters=response_converters),
)
try:
r = await attack.execute_async(objective="say something")
print(f"Part 2 response_converters={len(response_converters)} -> outcome={r.outcome.name}")
except Exception as e:
print(f"Part 2 response_converters={len(response_converters)} -> {type(e).__name__}: {str(e).splitlines()[0]}")
pieces = await CentralMemory.get_memory_instance().get_message_pieces_async()
print("Pieces in memory:", [(p.api_role, p.converted_value_data_type, p.response_error) for p in pieces])
asyncio.run(main())Expected Results
- An LLM-backed converter raises (or retries) when its target returns a blocked/error response, instead of returning the error payload as converted text.
- A blocked reply passes through response converters unchanged and is saved to memory, the same as when no response converters are configured.
Actual Results
Part 1 ToneConverter -> text {"status_code": 400, "message": "{\"error\": {\"code\": \"content_filter\", \"message\": \"The response was filtered\"}}"}
Part 2 response_converters=0 -> outcome=UNDETERMINED
Part 2 response_converters=1 -> _StrategyRuntimeError: Strategy execution failed for converter in PromptSendingAttack: Input type error not supported
Pieces in memory: [('user', 'text', 'none'), ('assistant', 'error', 'blocked'), ('user', 'text', 'none')]
The second run's user turn has no assistant reply in memory.
Possible fix
- Part 1: in
_send_with_retries_async, check the returned piece'sresponse_error(orerrordata type) and raise one of the exceptions inRETRY_EXCEPTIONS, so the existing retry path handles it. - Part 2: skip pieces with
response_error != "none"when applying response converters inPromptNormalizer.
I'm happy to open a PR with regression tests for both if this direction works for you.
Versions
- OS: macOS 27 (arm64)
- Python version: 3.13.11
- PyRIT version: 1.3.0.dev0, installed from main (
e24801a7d) in editable mode withuv sync --locked
AI assistance: an AI coding assistant (Claude Code) helped me find these code paths, write the reproduction and draft this report. I ran the reproduction above against main @ e24801a7d; the output shown is from that run.
Activity