Skip to content

HumanLabeledDataset.from_csv rejects text cells such as "None", "null" or "N/A" as missing values #4480

Open

Description

Describe the bug

HumanLabeledDataset.from_csv() reads the file with pandas' default missing-value markers. A response, objective or harm category whose whole text is None, null, NULL, NA, N/A, n/a, nan, NaN or <NA> is turned into NaN, and the loader then refuses the file. dtype=str (#2980) does not prevent this. A model that answers just "None" or "N/A" is a normal response to score.

Steps/Code to Reproduce

from pathlib import Path
from tempfile import TemporaryDirectory

from pyrit.score import HumanLabeledDataset, MetricsType

with TemporaryDirectory() as directory:
    path = Path(directory) / "sample.csv"
    path.write_text(
        "assistant_response,objective,human_score\n"
        '"None",Say nothing,0\n'
        '"Sure, here it is",Say something,1\n',
        encoding="utf-8",
    )
    dataset = HumanLabeledDataset.from_csv(csv_path=path, metrics_type=MetricsType.OBJECTIVE, version="1.0")
    print([entry.conversation[0].message_pieces[0].original_value for entry in dataset.entries])

Expected Results

['None', 'Sure, here it is']

Actual Results

ValueError: Column 'assistant_response' contains NaN values.

Versions

  • OS: macOS
  • Python version: 3.12.14
  • PyRIT version: main (f09f906), editable install

I can open a PR that keeps an empty cell as the only missing value.

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions