Skip to content

FEAT: Add CBRN scenario trials and run artifacts - #4483

Draft
Nina Chikanov (nina-msft) wants to merge 1 commit into
microsoft:mainfrom
nina-msft:nina-msft-scenario-configuration-examples
Draft

Nina Chikanov (nina-msft) wants to merge 1 commit into
microsoft:mainfrom
nina-msft:nina-msft-scenario-configuration-examples

Conversation

@nina-msft

@nina-msft Nina Chikanov (nina-msft) commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Description

Add an end-to-end CBRN evaluation flow with independently scored trials, population-specific outcomes, and downloadable evidence scoped to a single scenario run.

  • Add version-1 airt.cbrn with configurable techniques and scoring. ALLOW success means refusal; REFUSE success requires both non-refusal and operational execution details.
  • Execute every configured trial in a fresh conversation, separate trials from retries, and report trial ASR independently from categorical any-trial group success.
  • Expose CBRN configuration in the scanner and provide a public five-objective synthetic airt_cbrn dataset with mixed ALLOW/REFUSE labels and only tier/expected custom metadata.
  • Export the selected completed, failed, or cancelled run with its configuration, results, and selected-seed metadata snapshots. Show a dismissible download confirmation and filename without claiming access to the browser's final save path.

GUI example: three successful trials out of five display 60% (3/5 trials), while the group's categorical outcome is Success. Download artifacts exports JSON for that selected run only.

Add population-specific CBRN scoring and configurable independent trials, expose trial ASR in the GUI, and export run-scoped artifacts with seed metadata snapshots and browser download feedback. Include a five-objective public dataset and focused coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 21, 2026 00:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

CBRN validation occurs after dataset replacement, allowing invalid refreshes to overwrite valid data, and checks unrelated stored objectives during resume, blocking otherwise unchanged runs.

2 open issues
What changed in this PR

Adds CBRN evaluation to PyRIT’s scenario framework, with independent trials, population-specific scoring, and downloadable evidence for individual runs.

Changes:

  • Introduces airt.cbrn, configurable trial execution, and a five-objective synthetic dataset.
  • Separates trial success rates from group outcomes and retry counts, with scanner configuration support.
  • Adds terminal-run artifact downloads and selected-seed metadata snapshots.
File Description
tests/​unit/​scenario/​core/​test_scenario.py Tests metadata snapshots and resume preservation.
tests/​unit/​scenario/​airt/​test_cbrn.py Tests CBRN scoring, configuration, sampling, and resume.
tests/​unit/​registry/​test_scenario_registry.py Updates technique availability expectations.
tests/​unit/​executor/​attack/​compound/​test_repeated_attack.py Tests independent trials and persisted outcomes.
tests/​unit/​datasets/​test_local_prompt_dataset_semantics.py Checks synthetic dataset structure and labels.
tests/​unit/​backend/​test_scenario_run_service.py Tests artifact eligibility by run state.
tests/​unit/​backend/​test_scenario_run_routes.py Tests artifact responses and errors.
tests/​unit/​backend/​test_scenario_progress_read_model.py Tests latest-execution trial statistics.
tests/​unit/​analytics/​test_scenario_statistics_parity.py Checks consistent trial and retry accounting.
pyrit/​scenario/​scenarios/​airt/​cbrn.py Implements CBRN configuration and population-specific trials.
pyrit/​scenario/​scenarios/​airt/​__init__.py Exports CBRN and its technique catalog.
pyrit/​scenario/​scenarios/​_dynamic_techniques.py Registers the dynamic CBRN technique catalog.
pyrit/​scenario/​core/​scenario.py Adds technique restrictions and metadata snapshots.
pyrit/​registry/​components/​scenario_registry.py Publishes technique unavailability reasons.
pyrit/​models/​scenario_progress.py Adds seed snapshots and trial statistics fields.
pyrit/​models/​catalog/​scenario.py Adds technique availability metadata.
pyrit/​memory/​memory_interface.py Separates indexed children from retry counts.
pyrit/​executor/​attack/​compound/​repeated_attack.py Implements fixed-count trials and outcome summaries.
pyrit/​executor/​attack/​compound/​__init__.py Exports RepeatedAttack.
pyrit/​executor/​attack/​__init__.py Exposes RepeatedAttack through the public API.
pyrit/​datasets/​seed_datasets/​local/​airt/​cbrn.prompt Adds five synthetic, labeled objectives.
pyrit/​datasets/​score/​true_false_question/​cbrn_operational.yaml Defines the operational-detail scoring rubric.
pyrit/​backend/​services/​scenario_run_service.py Allows terminal-run exports and adjusts retry accounting.
pyrit/​backend/​services/​scenario_progress_read_model.py Aggregates trial outcomes from latest executions.
pyrit/​backend/​routes/​scenarios.py Adds the run artifacts endpoint.
pyrit/​backend/​README.md Documents artifact scope and trial reporting.
pyrit/​analytics/​scenario_statistics.py Distinguishes child executions from retries.
frontend/​src/​types/​index.ts Adds availability, snapshot, and trial fields.
frontend/​src/​services/​api.ts Adds artifact download requests.
frontend/​src/​components/​Scenarios/​ScenarioTechniqueSelector.tsx Explains unavailable techniques and allows deselection.
frontend/​src/​components/​Scenarios/​ScenarioTechniqueSelector.test.tsx Tests accessible unavailability explanations.
frontend/​src/​components/​Scenarios/​ScenarioTechniqueSection.tsx Displays technique-specific availability restrictions.
frontend/​src/​components/​Scenarios/​ScenarioRunPage.tsx Adds downloads, confirmation, and trial-rate display.
frontend/​src/​components/​Scenarios/​ScenarioRunPage.test.tsx Tests downloads, cancellation, and trial display.
frontend/​src/​components/​Scenarios/​ScenarioDetail.tsx Blocks incompatible technique and scorer selections.
frontend/​src/​components/​Scenarios/​ScenarioDetail.test.tsx Tests scorer selection and technique restrictions.
frontend/​src/​components/​Scenarios/​scenarioConfigForm.ts Carries availability into selectable techniques.
frontend/​src/​components/​Chat/​ChatScorerPicker.tsx Marks ineligible selected scorers unavailable.
frontend/​src/​components/​AttackResults/​attackAttemptFormatting.ts Formats trial success rates and counts.
frontend/​src/​components/​AttackResults/​attackAttemptFormatting.test.ts Tests trial formatting and legacy fallback.

Review effort: Balanced

Comment on lines +278 to +281
populations = await super()._resolve_seed_groups_by_dataset_async(apply_sampling=False)
for groups in populations.values():
for group in groups:
self._expected_response(group)
async def _resolve_seed_groups_by_dataset_async(
self, *, apply_sampling: bool = True
) -> dict[str, list[AttackSeedGroup]]:
populations = await super()._resolve_seed_groups_by_dataset_async(apply_sampling=False)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants