Skip to content

FEAT: Add CoSafe dataset loader (1,400 multi-turn coreference attacks) #4512

Open

Description

@romanlutz

Add a seed dataset loader for CoSafe, 1,400 multi-turn conversations in which the final harmful question refers back to earlier turns instead of naming the harmful thing directly.

Why it fits

A safety filter that looks at one message at a time can miss a request like "how do I make more of it?" when "it" was introduced two turns earlier. CoSafe tests exactly that: each conversation builds context over several turns and ends with a question that is only harmful once the reference is resolved.

Dataset facts

Source GitHub repository ErxinYu/CoSafe-Dataset
Licence None stated in the repository. The loader fetches the files at run time, like every remote loader; no data is added to PyRIT.
Access Open
CoSafe datasets/ 14 files, one per harm category (for example self_harm.json, privacy_violation.json), 100 conversations each
Single Prompt/ 14 files with the single-turn version of the same questions
Format Despite the .json extension, the multi-turn files are JSON Lines: each line is one conversation, a list of {"role": ..., "content": ...} messages. In the file I sampled, conversations alternate user and assistant and end on a user turn.
Paper Yu et al.: "CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference", EMNLP 2024, arXiv:2406.17626

Proposed shape

  • A CoSafeCategory enum for the 14 categories, and a constructor argument categories: list[CoSafeCategory] | None = None where None means all categories. Several file names contain commas; map each enum member to its exact file name.
  • One seed group per conversation: one SeedPrompt per message, with its role and an increasing sequence, all sharing a prompt_group_id. mask_dataset.py is an existing loader that builds multi-turn groups with roles this way.
  • harm_categories comes from the file the conversation was read from.
  • Fetch with _fetch_from_url. Check whether its JSON handling copes with JSON Lines content under a .json name, and handle that in the loader if it does not.
  • The Single Prompt/ files are out of scope unless you can show how they line up with the multi-turn conversations; if they do, the single-turn question belongs in metadata.

Implementation notes

  • Follow .github/instructions/datasets.instructions.md. It covers everything a loader needs: subclassing _RemoteDatasetLoader, choosing between SeedPrompt and SeedObjective, filter enums, per-seed metadata, class-level metadata, registration in remote/__init__.py, the BibTeX entry, the datasets doc page, and unit tests with mocked data.
  • aya_redteaming_dataset.py and tests/unit/datasets/test_aya_redteaming_dataset.py are a good template for both the loader and its tests.
  • The schema above was read from the dataset card. Run your loader once against the real dataset before opening the PR and confirm column names, category values and row counts; say in the PR description that you did.
  • Unit tests must mock the download. No test may fetch from the network.

Done when

  • The loader exists, is registered, and its dataset name appears in SeedDatasetProvider.get_all_dataset_names_async().
  • Unit tests cover the happy path, every filter value, the empty-after-filter ValueError, and invalid filter values.
  • The paper is cited in doc/references.bib and doc/bibliography.md, and the dataset is listed in doc/code/datasets/1_loading_datasets.
  • Unit tests and pre-commit pass.

How to contribute

  • Claim it first. Before you start, comment on this issue to say you are taking it.
  • First comment wins. If yours is the first claim comment, go ahead and open a PR; you do not need to wait for a reply. If someone has already commented, this issue is taken, so please pick a different one.
  • Link this issue in your PR description.

Activity

  1. Shizoqua commented on Nov 21, 2026

    @Shizoqua
    Contributor

    I'd like to work on this. Could you assign it to me?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    datasetsPulling in external datasets into PyRIThelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions