Repository navigation
Description
Describe the bug
SequenceCompletionPolicy.FIRST_SUCCESS is documented to "continue past other outcomes" as the "Resilient adaptive default", to "keep trying other strategies past transient errors", and EXHAUSTIVE as "Run every child attack" (sequential_attack.py, and the table in doc/code/executor/4_compound.py). In practice a child attack that fails never returns an ERROR result: the strategy persists the ERROR row and raises, and _run_child_attack_async re-raises the executor's incomplete-objective exception (sequential_attack.py#L447). So every policy stops at the first child that raises. The remaining children never run, and the objective is stored as ERROR.
The adaptive scenarios build each objective's attack as a SequentialAttack with FIRST_SUCCESS over up to max_attempts_per_objective (default 3) techniques (dispatcher.py#L374-L377). A transient error in the first technique, such as a 503 after exhausted retries or a scorer or converter error, therefore means the other techniques are never sent, and the objective counts as an ERROR unit in the scenario's success rate.
test_error_outcome_does_not_stop ("FIRST_SUCCESS is resilient: a transient ERROR should not abort the sequence") passes because the mocked children return an ERROR result, which real child attacks do not do.
#3039 kept exception propagation on purpose ("Do not turn thrown child exceptions into returned error results merely to retain links"), so this may be a question of which side to change: either FIRST_SUCCESS and EXHAUSTIVE catch a child's exception, record that child as ERROR (its result is already persisted and linked since #3039) and move on, or the policy docs and the adaptive scenarios' description should say that an error stops the sequence. I would be glad to send a PR for whichever you prefer.
Steps/Code to Reproduce
The target fails its first call and complies afterwards, so the second child would succeed:
import asyncio
from pyrit.executor.attack import (
AttackScoringConfig,
PromptSendingAttack,
SequenceCompletionPolicy,
SequentialAttack,
SequentialChildAttack,
)
from pyrit.models import AttackSeedGroup, Message, MessagePiece, SeedObjective
from pyrit.prompt_target import PromptTarget
from pyrit.score import SubStringScorer
from pyrit.setup import IN_MEMORY, initialize_pyrit_async
class FlakyTarget(PromptTarget):
"""Fails the first call, as after exhausted retries on a 503, then complies."""
calls = 0
async def _send_prompt_to_target_async(self, *, normalized_conversation: list[Message]) -> list[Message]:
self.calls += 1
if self.calls == 1:
raise RuntimeError("503 Service Unavailable")
conversation_id = normalized_conversation[-1].message_pieces[0].conversation_id
return [MessagePiece(role="assistant", original_value="Sure, ...", conversation_id=conversation_id).to_message()]
async def main() -> None:
await initialize_pyrit_async(memory_db_type=IN_MEMORY)
for policy in (SequenceCompletionPolicy.FIRST_SUCCESS, SequenceCompletionPolicy.EXHAUSTIVE):
objective = f"Explain how to pick a lock ({policy.value})"
target = FlakyTarget()
scoring = AttackScoringConfig(objective_scorer=SubStringScorer(substring="Sure", categories=["test"]))
children = [
SequentialChildAttack(
strategy=PromptSendingAttack(objective_target=target, attack_scoring_config=scoring),
seed_group=AttackSeedGroup(seeds=[SeedObjective(value=objective)]),
)
for _ in range(2)
]
attack = SequentialAttack(objective_target=target, child_attacks=children, completion_policy=policy)
try:
result = await attack.execute_async(objective=objective)
print(f"{policy.value}: outcome {result.outcome.value}, target calls {target.calls}")
except Exception as error:
print(f"{policy.value}: raised {type(error).__name__}, target calls {target.calls}")
asyncio.run(main())Expected Results
Per the policy docs, the first child is recorded as ERROR, the second child runs and succeeds, and the sequence's outcome is SUCCESS with 2 target calls, for both policies.
Actual Results
first_success: raised _StrategyRuntimeError, target calls 1
exhaustive: raised _StrategyRuntimeError, target calls 1
The second child succeeds when run on its own.
Versions
- OS: Linux
- Python version: 3.12
- PyRIT version: main at ce434d7
Activity
adimalkar commentedon Nov 18, 2026
This reproduces on
main(b613c6e) with real child attacks, not only the mocked ones intest_error_outcome_does_not_stop. The setup is twoPromptSendingAttackchildren: the first one's target raisesRuntimeError("simulated 503 after exhausted retries")from_send_prompt_to_target_async, and the second one's target answers normally (MockPromptTargetfromtests/unit/mocks.py,IN_MEMORYdatabase):So both policies stop at the first child that raises, and
AdaptiveTechniqueDispatcherstill builds its per-objective attack asSequentialAttackwithFIRST_SUCCESS.On the question of which side to change: the policy docstrings and the adaptive scenarios both rely on the resilient behaviour, so changing the behaviour seems closer to the intent. A narrow version would be:
_perform_async, catch an exception from_run_child_attack_asynconly underFIRST_SUCCESSandEXHAUSTIVE, record that child asERROR(its row is already persisted and linked), and move on to the next child.FIRST_DECISIVEandSTRICT_ALLkeep raising, as now.asyncio.CancelledError(and otherBaseExceptions) still propagate, so cancelling a scan isn't swallowed.ERROR. One open point is whether it should then re-raise the last child's exception instead of returning, so a fully failed objective still surfaces as an incomplete objective inAttackExecutor.ERRORresult.Siddhant Rajhans (@siddhant-rajhans), are you planning to send a PR for this? If not, and the maintainers agree on the behaviour change (and on the all-children-fail case), I can open one.
This was investigated with AI assistance (Claude Code).
siddhant-rajhans commentedon Nov 19, 2026
Thanks for reproducing it with real
PromptSendingAttackchildren. Yes, as offered in the issue, I plan to send the PR once a maintainer picks the direction. One thing for that choice: the behaviour change has to replace a #3039 test,test_parent_error_result_links_completed_and_failed_children, which runsEXHAUSTIVEwith a succeeding child and then a raising one and asserts that the child error still raises. When every child errors, I would re-raise the last child's exception, soAttackExecutorstill reports the objective as incomplete.romanlutz commentedon Nov 20, 2026
Thanks both — the repro with real children settles it. Let's change the behavior; the policy docs and the adaptive scenarios describe the intent, and the code doesn't match it.
Direction, essentially Aditya Malkar (@adimalkar)'s narrow version:
FIRST_SUCCESSandEXHAUSTIVE, catch a child'sException, record that child as ERROR (its row is already persisted and linked since FIX Preserve SequentialAttack child-result links when execution fails #3039), and continue with the next child.FIRST_DECISIVEandSTRICT_ALLkeep raising.asyncio.CancelledErrorand otherBaseExceptions must still propagate — cancelling a scan can't be swallowed.AttackExecutorstill reports the objective as incomplete; please attach a note (add_note) saying how many children failed, so the earlier failures aren't invisible.This doesn't contradict #3039: that decision was about not fabricating results just to keep links, and here nothing is fabricated — the child's ERROR row already exists. Replacing
test_parent_error_result_links_completed_and_failed_childrenis expected; please also fixtest_error_outcome_does_not_stopto use children that actually raise, and update the policy table indoc/code/executor/4_compound.pyto state these semantics.Siddhant Rajhans (@siddhant-rajhans) it's yours as offered — go ahead with the PR.
FIX: Continue past a raising child attack under FIRST_SUCCESS and EXH…