Skip to content

[Bug]: casting a float column to a signed integer type truncates on pandas and Polars, raises on PyArrow, and rounds on DuckDB #4019

Open

Description

@lucaluo925

Describe the bug

Casting a float column that holds non-integral values to an integer dtype gives three different outcomes depending on the backend:

  • pandas and Polars truncate toward zero
  • PyArrow (and pandas with the pyarrow dtype backend) raise ArrowInvalid: Float value 1.700000 was truncated converting to int64
  • DuckDB rounds half away from zero

So the same nw.col(...).cast(nw.Int64()) either returns 1, returns 2, or raises, for an input of 1.7.

The PyArrow side comes from src/narwhals/_arrow/series.py:563-565, which calls pc.cast(self.native, data_type) and so inherits PyArrow's default safe=True, under which a lossy float-to-int conversion is an error. Passing safe=False gives [1, -1, 2, -2, 0, 0] for the input below, matching pandas and Polars exactly. DuckDB would need an explicit truncation before the cast, which is a separate change.

Int64 and Int32 both show this. Unsigned targets are not affected in the same way: with a negative value in the data every backend raises, for the separate reason that the value is out of range.

Expr.cast and Series.cast do not document whether a lossy float-to-int cast truncates, rounds, or raises, and tests/expr_and_series/cast_test.py has no case with a non-integral float, which is presumably why this has not come up.

Steps or code to reproduce the bug

import duckdb
import pandas as pd
import polars as pl
import pyarrow as pa

import narwhals as nw

values = [1.7, -1.7, 2.5, -2.5, 0.9, -0.9]
data = {"idx": list(range(len(values))), "f": values}
pdf = pd.DataFrame(data)

frames = {
    "pandas": nw.from_native(pd.DataFrame(data)).lazy(),
    "pandas[pyarrow]": nw.from_native(
        pd.DataFrame(data).convert_dtypes(dtype_backend="pyarrow")
    ).lazy(),
    "polars": nw.from_native(pl.DataFrame(data)).lazy(),
    "pyarrow": nw.from_native(pa.table(data)).lazy(),
    "duckdb": nw.from_native(duckdb.sql("select * from pdf")),
}

for name, lf in frames.items():
    try:
        out = lf.select("idx", nw.col("f").cast(nw.Int64())).lazy().collect().sort("idx")
        print(f"{name:16} {out['f'].to_list()}")
    except Exception as exc:
        print(f"{name:16} {type(exc).__name__}: {exc}")

Expected results

One behaviour across backends. Taking Polars as the reference, truncation toward zero:

pandas           [1, -1, 2, -2, 0, 0]
pandas[pyarrow]  [1, -1, 2, -2, 0, 0]
polars           [1, -1, 2, -2, 0, 0]
pyarrow          [1, -1, 2, -2, 0, 0]
duckdb           [1, -1, 2, -2, 0, 0]

Actual results

pandas           [1, -1, 2, -2, 0, 0]
pandas[pyarrow]  ArrowInvalid: Float value 1.700000 was truncated converting to int64
polars           [1, -1, 2, -2, 0, 0]
pyarrow          ArrowInvalid: Float value 1.700000 was truncated converting to int64
duckdb           [2, -2, 2, -2, 1, -1]

Int32 behaves the same way:

pandas           [1, -1, 2, -2, 0, 0]
pandas[pyarrow]  ArrowInvalid: Float value 1.700000 was truncated converting to int32
polars           [1, -1, 2, -2, 0, 0]
pyarrow          ArrowInvalid: Float value 1.700000 was truncated converting to int32
duckdb           [2, -2, 2, -2, 1, -1]

Please run narwhals.show_versions() and enter the output below.

System:
    python: 3.14.7 (main, Aug  5 2026, 10:29:49) [Clang 21.0.0 (clang-2100.1.1.101)]
executable: /Users/luca/Projects/narwhals/.venv/bin/python
   machine: macOS-26.6.2-arm64-arm-64bit-Mach-O

Python dependencies:
     narwhals: 2.26.0
        numpy: 2.5.3
       pandas: 3.0.6
        modin:
         cudf:
      pyarrow: 25.0.1
      pyspark:
       polars: 1.44.2
         dask:
       duckdb: 1.5.6
         ibis:
     sqlframe:

Relevant log output

Traceback (most recent call last):
  ...
  File ".../narwhals/_arrow/series.py", line 565, in cast
    return self._with_native(pc.cast(self.native, data_type), preserve_broadcast=True)
  File ".../pyarrow/compute.py", line 414, in cast
    return call_function("cast", [arr], options, memory_pool)
  File "pyarrow/_compute.pyx", line 610, in pyarrow._compute.call_function
  File "pyarrow/_compute.pyx", line 405, in pyarrow._compute.Function.call
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Float value 1.700000 was truncated converting to int64

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions