Repository navigation
FIX: Continue past a raising child attack under FIRST_SUCCESS and EXHAUSTIVE - #4468
Open
Siddhant Rajhans (siddhant-rajhans) wants to merge 1 commit into
Conversation
…AUSTIVE SequentialAttack stopped at the first child attack that raised under every policy, although FIRST_SUCCESS is documented to keep trying past transient errors and EXHAUSTIVE to run every child. Under those two policies a child whose attack raises is now recorded as ERROR (its error result is already stored and linked) and the next child runs; FIRST_DECISIVE, STRICT_ALL and LAST_RESULT keep raising, and BaseExceptions such as cancellation propagate. Only a child whose attack raised (the strategy layer's _StrategyRuntimeError) and whose own error result was confirmed stored is passed over: a child that could not be prepared, or whose result could not be stored, still stops the sequence. If every child raises, the last child's exception is re-raised with a note counting the failures, so the objective is still reported as incomplete. Fixes microsoft#4238 Signed-off-by: Siddhant Rajhans <51508945+siddhant-rajhans@users.noreply.github.com>
Siddhant Rajhans (siddhant-rajhans)
force-pushed
the
fix/sequential-attack-child-errors
branch
from
November 20, 2026 09:57
8c3a8a7 to
dd151cc
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #4238, following the direction in this comment, which adopts the narrow version adimalkar proposed in the issue. Under
FIRST_SUCCESSandEXHAUSTIVE,SequentialAttacknow continues past a child attack that raises, as the policy docs and the adaptive scenarios describe._perform_asynccatches a child'sExceptionunderFIRST_SUCCESSandEXHAUSTIVE. It adds that child's ERROR result (already stored and linked since FIX Preserve SequentialAttack child-result links when execution fails #3039) tochild_attack_results, logs a warning, and runs the next child.FIRST_DECISIVEandSTRICT_ALLkeep raising, andasyncio.CancelledErrorand otherBaseExceptions are not caught.LAST_RESULTalso keeps raising. The direction didn't cover it, and its outcome is the last child's, so I left it as it was.It only continues past a child whose attack raised (the strategy layer's
_StrategyRuntimeError) and whose own ERROR result was confirmed stored. These still stop the sequence:ExceptionGroupthat carries the persistence error);This keeps the FIX Preserve SequentialAttack child-result links when execution fails #3039 uncertain-write and parameter-build tests unchanged. If the stored row cannot be read back, the error is logged and the child's own exception propagates.
If every child raises, the last child's exception is re-raised with an
add_notethat counts the failures and summarizes the earlier ones.AttackExecutorstill reports the objective as incomplete, and the parent's error result links every child.If some children raise and others complete, the normal aggregated result is returned with the failed children linked as ERROR. For example, ERROR then SUCCESS gives SUCCESS, and ERROR then FAILURE gives FAILURE.
The policy docstrings and the table in
doc/code/executor/4_compound.py(and its notebook) describe these semantics.One choice for you.
technique.executebuilds itsretries_on_objective_failureattempts as aFIRST_SUCCESSsequence. With this change, an attempt that raises is followed by the next attempt instead of leaving the objective incomplete, so the scenario'smax_retriesno longer handles that exception. Two tests from #3118 encoded the old split between the two retries, and I updated them (below).If you would rather keep exceptions on
max_retriesthere,Executecould build its sequence withFIRST_DECISIVE, which keeps both tests as they were. It would also stop on an attempt that returns an ERROR outcome rather than raising. I can switch to that.Tests
test_error_outcome_does_not_stopnow uses a child attack that raises (FIRST_SUCCESS: ERROR, then SUCCESS).test_parent_error_result_links_completed_and_failed_childrennow runs underSTRICT_ALLandLAST_RESULT, which still stop at a raising child. ItsEXHAUSTIVEcase is nowtest_exhaustive_runs_the_rest_and_links_the_failed_child.New
TestChildExceptionscovers these cases:FIRST_SUCCESS,EXHAUSTIVE);FIRST_DECISIVE,STRICT_ALLandLAST_RESULTstop at a raising child;In
test_sequential_attack.py, 5 of the new or changed tests fail onmain. With this change all 79 tests in that file pass.The two
technique.executetests from FEAT: Run one technique against an objective with technique.execute #3118:test_late_child_error_does_not_complete_the_compound_asyncis nowtest_child_error_is_recorded_and_the_other_attempts_run_async. A late error is recorded and the remaining attempts run, so there are 3 sends instead of 5. In the resume variant, every attempt of the first run now fails, so the objective stays incomplete and is resumed.test_error_retry_is_separate_from_objective_retry_async[1]now expects 2 sends, because the failed first attempt is followed by the second attempt instead of a scenario-level retry.test_failed_orchestration_parent_keeps_its_role_despite_having_a_conversation_id_asyncruns the adaptive-style group (two attempts) against a target that always fails. Both attempts now run and fail, so it expects two target-facing ERROR children (attempts 1 and 2) under the ERROR parent instead of one.On this commit,
pytest -n 4 tests/unit/executor tests/unit/scenario tests/unit/backend tests/unit/models tests/unit/registry tests/unit/analytics tests/unit/score/test_batch_scorer.py(every unit test area that usesSequentialAttack) gives 14,864 passed and 4 failed. Three are uvicorn lifespan tests inbackend/test_compatibility.pyandbackend/test_main.pythat fail the same way onmainin my environment. The fourth isregistry/test_converter_identifier_contract.py::test_identifiers_stay_the_same_in_another_process, which passes when run on its own. Pre-commit (ruff, the docs and notebook hooks) andtypass on the changed files.