Describe the bug
isnan currently uses math::isnan which propagates nulls and can mismatch Spark semantics. In Spark, isnan(null) must return false, not null. Additionally, computed NaN values like log(-3) may not be handled consistently during Parquet round-trip.
To Reproduce
Failing UTs as reported in #1595
create table test_is_nan using parquet as
select
cast('NaN' as double) as c1,
cast('NaN' as float) as c2,
log(-3) as c3,
cast(null as double) as c4,
5.5f as c5,
cast(null as float) as c6;
select isnan(c1), isnan(c2), isnan(c3), isnan(c4), isnan(c5), isnan(c6) from test_is_nan;
Expected behavior
isnan(c1) = true (NaN)
isnan(c2) = true (NaN)
isnan(c3) = true (NaN from log(-3)) [note: verify Parquet round-trip behavior]
isnan(c4) = false (null → false)
isnan(c5) = false
isnan(c6) = false (null → false)
Actual behavior
In native execution, nulls in isnan previously surfaced as:
java.lang.IllegalStateException: Value at index is null from spark_udf_wrapper.rs during project output.
Screenshots
Additional context