Skip to content

FIX: Continue past a raising child attack under FIRST_SUCCESS and EXHAUSTIVE - #4468

Open
Siddhant Rajhans (siddhant-rajhans) wants to merge 1 commit into
microsoft:mainfrom
siddhant-rajhans:fix/sequential-attack-child-errors
Open

Siddhant Rajhans (siddhant-rajhans) wants to merge 1 commit into
microsoft:mainfrom
siddhant-rajhans:fix/sequential-attack-child-errors

Conversation

@siddhant-rajhans

@siddhant-rajhans Siddhant Rajhans (siddhant-rajhans) commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

Fixes #4238, following the direction in this comment, which adopts the narrow version adimalkar proposed in the issue. Under FIRST_SUCCESS and EXHAUSTIVE, SequentialAttack now continues past a child attack that raises, as the policy docs and the adaptive scenarios describe.

  • _perform_async catches a child's Exception under FIRST_SUCCESS and EXHAUSTIVE. It adds that child's ERROR result (already stored and linked since FIX Preserve SequentialAttack child-result links when execution fails #3039) to child_attack_results, logs a warning, and runs the next child.

  • FIRST_DECISIVE and STRICT_ALL keep raising, and asyncio.CancelledError and other BaseExceptions are not caught. LAST_RESULT also keeps raising. The direction didn't cover it, and its outcome is the last child's, so I left it as it was.

  • It only continues past a child whose attack raised (the strategy layer's _StrategyRuntimeError) and whose own ERROR result was confirmed stored. These still stop the sequence:

    • a child that could not be prepared;
    • a child whose error result could not be stored (the ExceptionGroup that carries the persistence error);
    • a child whose completed result could not be stored, even one with an ERROR outcome.

    This keeps the FIX Preserve SequentialAttack child-result links when execution fails #3039 uncertain-write and parameter-build tests unchanged. If the stored row cannot be read back, the error is logged and the child's own exception propagates.

  • If every child raises, the last child's exception is re-raised with an add_note that counts the failures and summarizes the earlier ones. AttackExecutor still reports the objective as incomplete, and the parent's error result links every child.

  • If some children raise and others complete, the normal aggregated result is returned with the failed children linked as ERROR. For example, ERROR then SUCCESS gives SUCCESS, and ERROR then FAILURE gives FAILURE.

  • The policy docstrings and the table in doc/code/executor/4_compound.py (and its notebook) describe these semantics.

One choice for you. technique.execute builds its retries_on_objective_failure attempts as a FIRST_SUCCESS sequence. With this change, an attempt that raises is followed by the next attempt instead of leaving the objective incomplete, so the scenario's max_retries no longer handles that exception. Two tests from #3118 encoded the old split between the two retries, and I updated them (below).

If you would rather keep exceptions on max_retries there, Execute could build its sequence with FIRST_DECISIVE, which keeps both tests as they were. It would also stop on an attempt that returns an ERROR outcome rather than raising. I can switch to that.

Tests

  • test_error_outcome_does_not_stop now uses a child attack that raises (FIRST_SUCCESS: ERROR, then SUCCESS).

  • test_parent_error_result_links_completed_and_failed_children now runs under STRICT_ALL and LAST_RESULT, which still stop at a raising child. Its EXHAUSTIVE case is now test_exhaustive_runs_the_rest_and_links_the_failed_child.

  • New TestChildExceptions covers these cases:

    • when every child raises, the last error is re-raised with the note (FIRST_SUCCESS, EXHAUSTIVE);
    • FIRST_DECISIVE, STRICT_ALL and LAST_RESULT stop at a raising child;
    • cancellation is not swallowed;
    • a completed child with an ERROR result whose write fails still stops;
    • a failure without a confirmed error result still stops;
    • an unreadable error result propagates the child's error.

    In test_sequential_attack.py, 5 of the new or changed tests fail on main. With this change all 79 tests in that file pass.

  • The two technique.execute tests from FEAT: Run one technique against an objective with technique.execute #3118:

    • test_late_child_error_does_not_complete_the_compound_async is now test_child_error_is_recorded_and_the_other_attempts_run_async. A late error is recorded and the remaining attempts run, so there are 3 sends instead of 5. In the resume variant, every attempt of the first run now fails, so the objective stays incomplete and is resumed.
    • test_error_retry_is_separate_from_objective_retry_async[1] now expects 2 sends, because the failed first attempt is followed by the second attempt instead of a scenario-level retry.
  • test_failed_orchestration_parent_keeps_its_role_despite_having_a_conversation_id_async runs the adaptive-style group (two attempts) against a target that always fails. Both attempts now run and fail, so it expects two target-facing ERROR children (attempts 1 and 2) under the ERROR parent instead of one.

On this commit, pytest -n 4 tests/unit/executor tests/unit/scenario tests/unit/backend tests/unit/models tests/unit/registry tests/unit/analytics tests/unit/score/test_batch_scorer.py (every unit test area that uses SequentialAttack) gives 14,864 passed and 4 failed. Three are uvicorn lifespan tests in backend/test_compatibility.py and backend/test_main.py that fail the same way on main in my environment. The fourth is registry/test_converter_identifier_contract.py::test_identifiers_stay_the_same_in_another_process, which passes when run on its own. Pre-commit (ruff, the docs and notebook hooks) and ty pass on the changed files.

Copilot AI balanced review requested due to automatic review settings November 20, 2026 09:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

…AUSTIVE

SequentialAttack stopped at the first child attack that raised under every
policy, although FIRST_SUCCESS is documented to keep trying past transient
errors and EXHAUSTIVE to run every child. Under those two policies a child
whose attack raises is now recorded as ERROR (its error result is already
stored and linked) and the next child runs; FIRST_DECISIVE, STRICT_ALL and
LAST_RESULT keep raising, and BaseExceptions such as cancellation propagate.

Only a child whose attack raised (the strategy layer's _StrategyRuntimeError)
and whose own error result was confirmed stored is passed over: a child that
could not be prepared, or whose result could not be stored, still stops the
sequence. If every child raises, the last child's exception is re-raised with
a note counting the failures, so the objective is still reported as
incomplete.

Fixes microsoft#4238

Signed-off-by: Siddhant Rajhans <51508945+siddhant-rajhans@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: SequentialAttack stops at the first child attack that raises, so FIRST_SUCCESS and EXHAUSTIVE never try the rest

2 participants