Skip to content

FEAT: Add MultiJail dataset loader - #4510

Open
Manikanta Boda (LuckysHorizon) wants to merge 1 commit into
microsoft:mainfrom
LuckysHorizon:feat/multijail-dataset-loader
Open

Manikanta Boda (LuckysHorizon) wants to merge 1 commit into
microsoft:mainfrom
LuckysHorizon:feat/multijail-dataset-loader

Conversation

@LuckysHorizon

Copy link
Copy Markdown

Description

Adds a seed dataset loader for MultiJail, 315 unsafe prompts human-translated into ten languages, so a target's behavior can be compared across languages on identical content. Closes #4494.

  • Adds _MultiJailDataset and a MultiJailLanguage enum; languages=None loads all ten languages.
  • Produces one SeedPrompt per row and selected language, with the language, source row id and source in metadata so prompts can be matched across languages.
  • Reads harm_categories from tags (Python-list strings in the real data), keeping the source casing.
  • Registers the loader and enum in remote/__init__.py, and adds the citation and datasets doc entry.
  • Ran the loader against the real Hugging Face dataset: 315 rows, 3150 seeds by default, 315 per language.

Copilot AI balanced review requested due to automatic review settings November 21, 2026 07:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the MultiJail dataset integration.

No open issues 🎉

What changed in this PR

Adds MultiJail to PyRIT’s seed dataset framework, enabling comparisons of target behavior across ten human-translated language versions of the same prompts.

Changes:

  • Adds a loader with language filtering, harm-category parsing, and metadata for matching prompts across languages.
  • Registers the loader and language enum, and adds mocked unit tests for filtering, metadata, and empty or invalid inputs.
  • Updates dataset documentation and adds the paper citation.
File Description
tests/​unit/​datasets/​test_multijail_dataset.py Tests language selection, seed metadata, tag parsing, and validation.
pyrit/​datasets/​seed_datasets/​remote/​multijail_dataset.py Implements the MultiJail loader and language enum.
pyrit/​datasets/​seed_datasets/​remote/​__init__.py Adds lazy exports for the loader and enum.
doc/​references.bib Adds the MultiJail paper reference.
doc/​code/​datasets/​1_loading_datasets.py Lists MultiJail in the dataset guide.
doc/​code/​datasets/​1_loading_datasets.ipynb Updates the guide and displayed dataset-name list.
doc/​bibliography.md Adds the paper’s citation key.

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FEAT: Add MultiJail dataset loader (315 unsafe prompts in 10 languages)

3 participants