Repository navigation
Description
Describe the bug
Casting a float column that holds non-integral values to an integer dtype gives three different outcomes depending on the backend:
- pandas and Polars truncate toward zero
- PyArrow (and pandas with the pyarrow dtype backend) raise
ArrowInvalid: Float value 1.700000 was truncated converting to int64 - DuckDB rounds half away from zero
So the same nw.col(...).cast(nw.Int64()) either returns 1, returns 2, or raises, for an input of 1.7.
The PyArrow side comes from src/narwhals/_arrow/series.py:563-565, which calls pc.cast(self.native, data_type) and so inherits PyArrow's default safe=True, under which a lossy float-to-int conversion is an error. Passing safe=False gives [1, -1, 2, -2, 0, 0] for the input below, matching pandas and Polars exactly. DuckDB would need an explicit truncation before the cast, which is a separate change.
Int64 and Int32 both show this. Unsigned targets are not affected in the same way: with a negative value in the data every backend raises, for the separate reason that the value is out of range.
Expr.cast and Series.cast do not document whether a lossy float-to-int cast truncates, rounds, or raises, and tests/expr_and_series/cast_test.py has no case with a non-integral float, which is presumably why this has not come up.
Steps or code to reproduce the bug
import duckdb
import pandas as pd
import polars as pl
import pyarrow as pa
import narwhals as nw
values = [1.7, -1.7, 2.5, -2.5, 0.9, -0.9]
data = {"idx": list(range(len(values))), "f": values}
pdf = pd.DataFrame(data)
frames = {
"pandas": nw.from_native(pd.DataFrame(data)).lazy(),
"pandas[pyarrow]": nw.from_native(
pd.DataFrame(data).convert_dtypes(dtype_backend="pyarrow")
).lazy(),
"polars": nw.from_native(pl.DataFrame(data)).lazy(),
"pyarrow": nw.from_native(pa.table(data)).lazy(),
"duckdb": nw.from_native(duckdb.sql("select * from pdf")),
}
for name, lf in frames.items():
try:
out = lf.select("idx", nw.col("f").cast(nw.Int64())).lazy().collect().sort("idx")
print(f"{name:16} {out['f'].to_list()}")
except Exception as exc:
print(f"{name:16} {type(exc).__name__}: {exc}")Expected results
One behaviour across backends. Taking Polars as the reference, truncation toward zero:
pandas [1, -1, 2, -2, 0, 0]
pandas[pyarrow] [1, -1, 2, -2, 0, 0]
polars [1, -1, 2, -2, 0, 0]
pyarrow [1, -1, 2, -2, 0, 0]
duckdb [1, -1, 2, -2, 0, 0]
Actual results
pandas [1, -1, 2, -2, 0, 0]
pandas[pyarrow] ArrowInvalid: Float value 1.700000 was truncated converting to int64
polars [1, -1, 2, -2, 0, 0]
pyarrow ArrowInvalid: Float value 1.700000 was truncated converting to int64
duckdb [2, -2, 2, -2, 1, -1]
Int32 behaves the same way:
pandas [1, -1, 2, -2, 0, 0]
pandas[pyarrow] ArrowInvalid: Float value 1.700000 was truncated converting to int32
polars [1, -1, 2, -2, 0, 0]
pyarrow ArrowInvalid: Float value 1.700000 was truncated converting to int32
duckdb [2, -2, 2, -2, 1, -1]
Please run narwhals.show_versions() and enter the output below.
System:
python: 3.14.7 (main, Aug 5 2026, 10:29:49) [Clang 21.0.0 (clang-2100.1.1.101)]
executable: /Users/luca/Projects/narwhals/.venv/bin/python
machine: macOS-26.6.2-arm64-arm-64bit-Mach-O
Python dependencies:
narwhals: 2.26.0
numpy: 2.5.3
pandas: 3.0.6
modin:
cudf:
pyarrow: 25.0.1
pyspark:
polars: 1.44.2
dask:
duckdb: 1.5.6
ibis:
sqlframe:Relevant log output
Traceback (most recent call last):
...
File ".../narwhals/_arrow/series.py", line 565, in cast
return self._with_native(pc.cast(self.native, data_type), preserve_broadcast=True)
File ".../pyarrow/compute.py", line 414, in cast
return call_function("cast", [arr], options, memory_pool)
File "pyarrow/_compute.pyx", line 610, in pyarrow._compute.call_function
File "pyarrow/_compute.pyx", line 405, in pyarrow._compute.Function.call
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Float value 1.700000 was truncated converting to int64
Activity