Skip to content

FIX: Separate baseline outcomes from attack reporting - #4484

Open
oliarmak-web wants to merge 2 commits into
microsoft:mainfrom
oliarmak-web:fix/baseline-reporting-4408
Open

oliarmak-web wants to merge 2 commits into
microsoft:mainfrom
oliarmak-web:fix/baseline-reporting-4408

Conversation

@oliarmak-web

Copy link
Copy Markdown

Description

Scenario summaries currently include baseline/control outcomes in attack success rates. This separates explicitly recorded baselines in pretty and JSON scenario/suite reports while retaining every per-group breakdown. One successful attack plus one failed baseline now reports attack success 100% and baseline success 0%, rather than a mixed 50%.

  • Select populations using saved run-plan group identities and kinds, including configurations sharing an atomic name; names alone never establish a baseline. Legacy results without recorded kinds keep historical counts.
  • Exclude baseline executions and attempts from scenario attack totals and suite-entry attack statistics, and expose baseline counts/rates separately. Baseline-only suite entries have no attack success percentage. Stored results, SDK analytics and scheduling are unchanged.
  • Add deterministic mixed, shared-name, baseline-only, legacy and nonmutation regression coverage and paired output documentation. All 536 relevant output/analytics tests pass; changed Python passes Ruff and ty. Tests use no models or credentials. Full unit suite and live notebook execution are not claimed; the notebook change is explanatory text only.

Addresses the reporting portion of #4408, as agreed in #4408 (comment) and the preceding clarification. Please keep #4408 open for suite-wide baseline scheduling.

AI assistance: OpenAI Codex implemented and tested this change under the account owner's direction.

Copilot AI balanced review requested due to automatic review settings November 21, 2026 01:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Report filtering can incorrectly attribute unmatched rows with shared configuration names and missing parent hashes to attack completions, changing success rates; full-plan resolution and regression coverage are needed.

1 open issue
What changed in this PR

This PR addresses the reporting portion of #4408 by separating recorded baseline outcomes from attack statistics in PyRIT’s scenario and suite reports. Suite-wide baseline scheduling remains a follow-up.

Changes:

  • Selects reporting populations using saved group identities and kinds, preserving legacy behavior when kinds are absent.
  • Adds separate baseline statistics to pretty and JSON summaries while retaining per-group breakdowns.
  • Adds regression coverage and paired documentation without changing stored results, SDK analytics, or execution.
File Description
tests/​unit/​output/​test_baseline_reporting.py Covers mixed, shared-name, baseline-only, legacy, and nonmutation cases.
pyrit/​output/​scenario_suite_result/​pretty.py Displays separate baseline counts and success rates.
pyrit/​output/​scenario_suite_result/​json.py Adds baseline statistics to suite-entry JSON.
pyrit/​output/​scenario_suite_result/​base.py Carries baseline statistics into suite report entries.
pyrit/​output/​scenario_result/​pretty.py Separates baseline statistics from overall attack totals.
pyrit/​output/​scenario_result/​json.py Separates attack totals and baseline statistics in JSON.
pyrit/​output/​_derivation.py Adds population filtering and separate statistics derivation.
doc/​code/​output/​0_output.py Documents reporting semantics and legacy behavior.
doc/​code/​output/​0_output.ipynb Adds matching explanatory text to the paired notebook.

Review effort: Balanced

kept.append(row)
if kept or any(g.atomic_attack_name == name and g.id in selected for g in plan.atomic_groups):
attacks[name] = kept
filtered_plan = plan.model_copy(update={"atomic_groups": [g for g in plan.atomic_groups if g.id in selected]})
@oliarmak-web

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants