Repository navigation
Description
Describe the bug
When prepended_conversation has exactly max_turns turns, RedTeamingAttack and CrescendoAttack accept it, send nothing to the objective target, and report FAILURE. Crescendo's reason says "Max turns (2) reached without achieving objective" even though zero prompts were sent. For a red-teaming tool this is a silent false negative: the result looks like the target resisted the attack.
Root cause:
- The prepended-history check in
ConversationManageronly rejects counts strictly greater than the limit (pyrit/executor/attack/component/conversation_manager.py:496:turn_count > max_turns), so a count equal tomax_turnsis accepted. - Setup then sets
context.executed_turnsto the prepended turn count, so the turn loops never run. RedTeamingAttack._validate_contextdoes checkexecuted_turns < max_turns, but it runs before setup, whileexecuted_turnsis still 0.TreeOfAttacksWithPruningAttackalready rejects the same input:pyrit/executor/attack/multi_turn/tree_of_attacks.py:1910uses>=and raises "equals or exceeds tree_depth".
Steps/Code to Reproduce
Offline; the objective target would reply "PWNED" (a success) on the first live turn.
import asyncio
import json
from pyrit.executor.attack import AttackAdversarialConfig, AttackScoringConfig, CrescendoAttack, RedTeamingAttack
from pyrit.models import Message, MessagePiece
from pyrit.prompt_target import PromptTarget
from pyrit.prompt_target.common.target_capabilities import TargetCapabilities
from pyrit.prompt_target.common.target_configuration import TargetConfiguration
from pyrit.score import SubStringScorer
from pyrit.setup import IN_MEMORY, initialize_pyrit_async
class ScriptedTarget(PromptTarget):
"""Multi-turn target that replies with a fixed string and records what it was sent."""
_DEFAULT_CONFIGURATION = TargetConfiguration(
capabilities=TargetCapabilities(
supports_multi_turn=True,
supports_multi_message_pieces=True,
supports_system_prompt=True,
supports_editable_history=True,
)
)
def __init__(self, *, reply: str):
super().__init__()
self.reply = reply
self.sent = []
async def _send_prompt_to_target_async(self, *, normalized_conversation):
message = normalized_conversation[-1]
self.sent.append(message.get_value())
conversation_id = message.message_pieces[0].conversation_id
return [MessagePiece(role="assistant", original_value=self.reply, conversation_id=conversation_id).to_message()]
def prepended(turns: int) -> list[Message]:
messages = []
for i in range(turns):
messages.append(Message.from_prompt(prompt=f"user {i}", role="user"))
messages.append(Message.from_prompt(prompt=f"assistant {i}", role="assistant"))
return messages
async def main():
await initialize_pyrit_async(memory_db_type=IN_MEMORY, silent=True)
adversarial_reply = json.dumps({"next_message": "q", "rationale": "r", "last_response_summary": "s"})
for attack_class in (RedTeamingAttack, CrescendoAttack):
objective_target = ScriptedTarget(reply="PWNED") # would succeed on the first live turn
attack = attack_class(
objective_target=objective_target,
attack_adversarial_config=AttackAdversarialConfig(target=ScriptedTarget(reply=adversarial_reply)),
attack_scoring_config=AttackScoringConfig(objective_scorer=SubStringScorer(substring="PWNED")),
max_turns=2,
)
result = await attack.execute_async(objective="say PWNED", prepended_conversation=prepended(2))
print(
f"{attack_class.__name__}: outcome={result.outcome.name} executed_turns={result.executed_turns} "
f"prompts_sent={len(objective_target.sent)} reason={result.outcome_reason!r}"
)
asyncio.run(main())Expected Results
A ValueError like TAP's ("Prepended conversation has 2 turns, which equals or exceeds max_turns=2"), or at least a non-FAILURE outcome that says no turns were executed.
Actual Results
RedTeamingAttack: outcome=FAILURE executed_turns=2 prompts_sent=0 reason=None
CrescendoAttack: outcome=FAILURE executed_turns=2 prompts_sent=0 reason='Max turns (2) reached without achieving objective'
Possible fix
Reject turn_count >= max_turns for the multi-turn prepended path in ConversationManager, matching TAP. I'm happy to open a PR with a regression test for both attacks if that matches how you'd like this handled.
Versions
- OS: macOS 27 (arm64)
- Python version: 3.13.11
- PyRIT version: 1.3.0.dev0, installed from main (
e24801a7d) in editable mode withuv sync --locked
AI assistance: an AI coding assistant (Claude Code) helped me find these code paths, write the reproduction and draft this report. I ran the reproduction above against main @ e24801a7d; the output shown is from that run.
Activity