Repository navigation
Description
Trying LLMFileAnalysisOperator on a small CSV on main (5114080, common.ai 1.1.0, macOS arm64, Python 3.12), the model gets the sample rows with their quotes removed. A value with a comma in it then looks like two values:
from pathlib import Path
from airflow.providers.common.ai.utils.file_analysis import build_file_analysis_request
Path("people.csv").write_text('name,city,amount\n"Smith, John","Vancouver, BC",120\nAlice,Toronto,80\n')
request = build_file_analysis_request(
file_path="people.csv", file_conn_id=None, prompt="Total amount per city?",
multi_modal=False, max_files=20, max_file_size_bytes=5 * 1024 * 1024,
max_total_size_bytes=20 * 1024 * 1024, max_text_chars=100_000, sample_rows=10,
)
print(request.user_content)Header: name, city, amount
Sample rows:
Smith, John, Vancouver, BC, 120
Alice, Toronto, 80
The header has 3 columns and the first row now shows 5 values, so a question like "total amount per city" has nothing reliable to work from. A quoted value with a line break in it is split across two lines the same way. This is the text execute() sends as the prompt:
airflow/providers/common/ai/src/airflow/providers/common/ai/operators/llm_file_analysis.py
Line 142 in 5114080
| request = build_file_analysis_request( |
_render_csv reads the file with csv.reader, which removes the quotes, then joins each row back with ", ":
airflow/providers/common/ai/src/airflow/providers/common/ai/utils/file_analysis.py
Lines 485 to 488 in 5114080
| payload = ["Header: " + ", ".join(header)] | |
| if sampled_rows: | |
| payload.append("Sample rows:") | |
| payload += [", ".join(str(value) for value in row) for row in sampled_rows] |
JSON, Parquet and Avro samples go through dumps_masked, so they keep their structure (Parquet, for example: L543). CSV is the only one that loses it.
What I'd suggest: quote a value the way CSV does when it holds a comma, a quote or a line break, and leave the rest alone, so the current alpha, 1 format stays the same:
def _join_csv_values(values: list[str]) -> str:
quoted = [
'"' + value.replace('"', '""') + '"' if any(char in value for char in ',"\r\n') else value
for value in values
]
return ", ".join(quoted)The first row then reads "Smith, John", "Vancouver, BC", 120.
I tried this locally:
- a new test in
test_file_analysis.py(a quoted comma, an escaped quote and a line break) fails on main and passes with the change; test_file_analysis.py,test_llm_file_analysis.pyandtest_object_storage.pypass (120 passed, 8 skipped because fastavro isn't installed);- running the operator itself with a
FunctionModelthat records its prompt shows the same thing before and after.
What should happen instead: each CSV value should reach the model intact, so the sample rows have the same columns as the header.
I didn't find an existing issue or PR for this. I have the fix and the test ready and will open a PR.
Drafted-by: Claude Code (Opus 5.5); reviewed by @aarmens702-hub before posting
Activity
boring-cyborg commentedon Nov 3, 2026
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.