Skip to content

LLMFileAnalysisOperator drops CSV quoting, so values with commas look like extra columns #76386

Closed

Description

@aarmens702-hub

Trying LLMFileAnalysisOperator on a small CSV on main (5114080, common.ai 1.1.0, macOS arm64, Python 3.12), the model gets the sample rows with their quotes removed. A value with a comma in it then looks like two values:

from pathlib import Path

from airflow.providers.common.ai.utils.file_analysis import build_file_analysis_request

Path("people.csv").write_text('name,city,amount\n"Smith, John","Vancouver, BC",120\nAlice,Toronto,80\n')
request = build_file_analysis_request(
    file_path="people.csv", file_conn_id=None, prompt="Total amount per city?",
    multi_modal=False, max_files=20, max_file_size_bytes=5 * 1024 * 1024,
    max_total_size_bytes=20 * 1024 * 1024, max_text_chars=100_000, sample_rows=10,
)
print(request.user_content)
Header: name, city, amount
Sample rows:
Smith, John, Vancouver, BC, 120
Alice, Toronto, 80

The header has 3 columns and the first row now shows 5 values, so a question like "total amount per city" has nothing reliable to work from. A quoted value with a line break in it is split across two lines the same way. This is the text execute() sends as the prompt:

_render_csv reads the file with csv.reader, which removes the quotes, then joins each row back with ", ":

payload = ["Header: " + ", ".join(header)]
if sampled_rows:
payload.append("Sample rows:")
payload += [", ".join(str(value) for value in row) for row in sampled_rows]

JSON, Parquet and Avro samples go through dumps_masked, so they keep their structure (Parquet, for example: L543). CSV is the only one that loses it.

What I'd suggest: quote a value the way CSV does when it holds a comma, a quote or a line break, and leave the rest alone, so the current alpha, 1 format stays the same:

def _join_csv_values(values: list[str]) -> str:
    quoted = [
        '"' + value.replace('"', '""') + '"' if any(char in value for char in ',"\r\n') else value
        for value in values
    ]
    return ", ".join(quoted)

The first row then reads "Smith, John", "Vancouver, BC", 120.

I tried this locally:

  • a new test in test_file_analysis.py (a quoted comma, an escaped quote and a line break) fails on main and passes with the change;
  • test_file_analysis.py, test_llm_file_analysis.py and test_object_storage.py pass (120 passed, 8 skipped because fastavro isn't installed);
  • running the operator itself with a FunctionModel that records its prompt shows the same thing before and after.

What should happen instead: each CSV value should reach the model intact, so the sample rows have the same columns as the header.

I didn't find an existing issue or PR for this. I have the fix and the test ready and will open a PR.


Drafted-by: Claude Code (Opus 5.5); reviewed by @aarmens702-hub before posting

Activity

  1. boring-cyborg commented on Nov 3, 2026

    @boring-cyborg

    Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions