Skip to content

[C++][Python] tdigest and hash_tdigest return NaN for q outside [0, 1]; quantile raises #55372

Open

Description

@chrikrah

Describe the bug, including details regarding any error messages, version, and platform.

pyarrow.compute.tdigest and hash_tdigest accept a q outside [0, 1] and return NaN, while quantile rejects the same value with ArrowInvalid. TDigestOptions documents the range as a requirement.

import pyarrow as pa
import pyarrow.compute as pc

print(pa.__version__)
a = pa.array([1.0, 2.0, 3.0])
for fn in (pc.quantile, pc.tdigest):
    try:
        print(fn.__name__, fn(a, q=[0.5, 1.5]).to_pylist())
    except pa.ArrowInvalid as e:
        print(fn.__name__, type(e).__name__, e)

t = pa.table({"k": [1, 1, 1], "v": [1.0, 2.0, 3.0]})
agg = t.group_by("k").aggregate([("v", "tdigest", pc.TDigestOptions(q=[7.0]))])
print(agg.to_pylist())
$ python -I td.py    # pyarrow 26.0.0 wheel, Python 3.12, Linux x86_64
26.0.0
quantile ArrowInvalid Quantile must be between 0 and 1
tdigest [2.0, nan]
[{'k': 1, 'v_tdigest': [nan]}]

contract: TDigestOptions.q in python/pyarrow/_compute.pyx:2388 ("All values must be in [0, 1]"), and cpp/src/arrow/compute/api_aggregate.h ("probability level of quantile must be between 0 and 1 inclusive").

CheckQuantileOptions (cpp/src/arrow/compute/kernels/aggregate_quantile.cc:482, a99464f on main) returns Status::Invalid("Quantile must be between 0 and 1") for any q outside the range. Neither aggregate_tdigest.cc nor GroupedTDigestImpl in hash_aggregate_numeric.cc (the separate hash_tdigest kernel) checks q, and TDigest::Quantile in cpp/src/arrow/util/tdigest.cc returns NAN for q < 0 || q > 1.

Expected: ArrowInvalid for any q outside [0, 1], from a shared check called by both the tdigest and hash_tdigest inits, with the message quantile uses.

I read the source at a99464f and ran the 26.0.0 wheel, not a source build. I did not check how q=NaN should behave: quantile also lets it through.

I can open a PR with the check and tests in test_compute.py, kernels/aggregate_test.cc and acero/hash_aggregate_test.cc. Would you take it?

Component(s)

C++, Python

Activity

  1. Abhi2939 commented on Nov 4, 2026

    @Abhi2939

    Hi @chrikrah, this looks like a clean fix. Are you still planning to open the PR? If you’d rather not, I’d be happy to take it. I’d add the check next to CheckQuantileOptions, call it from both the tdigest and hash_tdigest inits with the same message, and add tests in the three places you listed. I’ll leave q=NaN as it is now unless a maintainer wants it changed.

  2. chrikrah commented on Nov 6, 2026

    @chrikrah
    Author

    Yes, it is up as #55409. It also rejects q=NaN, which aborts a Debug build. Thanks for the offer, @Abhi2939.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions