Repository navigation
FIX: Separate baseline outcomes from attack reporting - #4484
Open
oliarmak-web wants to merge 2 commits into
Open
oliarmak-web wants to merge 2 commits into
oliarmak-web wants to merge 2 commits into
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Report filtering can incorrectly attribute unmatched rows with shared configuration names and missing parent hashes to attack completions, changing success rates; full-plan resolution and regression coverage are needed.
1 open issue
What changed in this PR
This PR addresses the reporting portion of #4408 by separating recorded baseline outcomes from attack statistics in PyRIT’s scenario and suite reports. Suite-wide baseline scheduling remains a follow-up.
Changes:
- Selects reporting populations using saved group identities and kinds, preserving legacy behavior when kinds are absent.
- Adds separate baseline statistics to pretty and JSON summaries while retaining per-group breakdowns.
- Adds regression coverage and paired documentation without changing stored results, SDK analytics, or execution.
| File | Description |
|---|---|
tests/unit/output/test_baseline_reporting.py |
Covers mixed, shared-name, baseline-only, legacy, and nonmutation cases. |
pyrit/output/scenario_suite_result/pretty.py |
Displays separate baseline counts and success rates. |
pyrit/output/scenario_suite_result/json.py |
Adds baseline statistics to suite-entry JSON. |
pyrit/output/scenario_suite_result/base.py |
Carries baseline statistics into suite report entries. |
pyrit/output/scenario_result/pretty.py |
Separates baseline statistics from overall attack totals. |
pyrit/output/scenario_result/json.py |
Separates attack totals and baseline statistics in JSON. |
pyrit/output/_derivation.py |
Adds population filtering and separate statistics derivation. |
doc/code/output/0_output.py |
Documents reporting semantics and legacy behavior. |
doc/code/output/0_output.ipynb |
Adds matching explanatory text to the paired notebook. |
Review effort: Balanced
| kept.append(row) | ||
| if kept or any(g.atomic_attack_name == name and g.id in selected for g in plan.atomic_groups): | ||
| attacks[name] = kept | ||
| filtered_plan = plan.model_copy(update={"atomic_groups": [g for g in plan.atomic_groups if g.id in selected]}) |
Author
|
@microsoft-github-policy-service agree |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Description
Scenario summaries currently include baseline/control outcomes in attack success rates. This separates explicitly recorded baselines in pretty and JSON scenario/suite reports while retaining every per-group breakdown. One successful attack plus one failed baseline now reports attack success 100% and baseline success 0%, rather than a mixed 50%.
Addresses the reporting portion of #4408, as agreed in #4408 (comment) and the preceding clarification. Please keep #4408 open for suite-wide baseline scheduling.
AI assistance: OpenAI Codex implemented and tested this change under the account owner's direction.