Skip to content

TYP: allow scalar values in DataFrame flexible comparison methods - #1979

Open
ZMZM6max wants to merge 1 commit into
pandas-dev:mainfrom
ZMZM6max:fix/dframe-flex-comp-scalar
Open

ZMZM6max wants to merge 1 commit into
pandas-dev:mainfrom
ZMZM6max:fix/dframe-flex-comp-scalar

Conversation

@ZMZM6max

Copy link
Copy Markdown

Annotates other as Scalar | ListLike | DataFrame in the six DataFrame flexible comparison methods, matching the documented behavior ("scalar, sequence, Series, or DataFrame") and the existing Series.eq annotation. Arithmetic methods are unchanged, since e.g. df + "foo" is not valid at runtime.

Tests cover str, int, float, bool, Timestamp, Timedelta and datetime scalar operands plus list-like, Series and DataFrame operands, and guard against a non-scalar object.

Validated with poe test: mypy, pyright, pyrefly, pytest (3409 passed) and style all clean; ty reports only the six pre-existing diagnostics on main from ty 0.0.86 (none in the files touched here).

AI Implementation Plan

Per AGENTS.md, a comprehensive implementation record for agents:

Motivation: #1978 — DataFrame.eq/ne/le/lt/ge/gt typed other: complex | ListLike | DataFrame, rejecting valid runtime scalars (str, bool, Timestamp, ...). The docs describe other as "scalar, sequence, Series, or DataFrame" and Series.eq already uses Scalar | Series[S1].

Approach: replace complex with Scalar in exactly the six comparison signatures in pandas-stubs/core/frame.pyi. Scalar is a strict superset of complex (adds the _IndexIterScalar members: str, bytes, date/time types, bool, int, float, Timestamp, Timedelta, plus np scalars), so the change is backward compatible.

Experiments and attempts:

  • Probed 20 public-API snippets against main; only the flexible comparisons rejected valid runtime input; df.eq(["foo", 1]), df.eq(df) etc. were already accepted and are kept as regression tests.
  • Initial test draft compared a mixed-dtype frame with lt("m"); pandas raises TypeError: Invalid comparison between dtype=int64 and str at runtime (ordering comparisons do not fall back to all-False like equality). Fixed by testing ordering comparisons against homogeneous string frames and numeric comparisons against numeric frames.
  • The invalid-usage guard uses object(), which is outside Scalar | ListLike | DataFrame for all four checkers.

Caveats: Scalar is wider than the set of dtypes a given frame can meaningfully compare with; e.g. df_num.lt("m") type-checks but raises at runtime. This matches the Series side and the docs, which defer elementwise comparison semantics to runtime.

Validation: poe mypy/pyright/pyrefly/pytest/style all pass (pytest: 3409 passed, 5 skipped). poe ty fails with the same six invalid-property-type-override diagnostics as pristine main (ty 0.0.86, arrays/*.pyi only; unrelated to this change).

Out of scope: widening arithmetic operators (invalid at runtime); Series flexible comparisons (already accept scalars); fill_value typing for string dtypes (tracked in #1797).

Co-Authored-By: Claude Code <noreply@anthropic.com>
Comment thread tests/frame/test_frame.py
Comment on lines +4639 to +4672
def test_flexible_comparisons_scalar() -> None:
"""Test DataFrame flexible comparison methods accept scalar values."""
df_str = pd.DataFrame({"A": ["foo", "bar"]})
check(assert_type(df_str.eq("foo"), pd.DataFrame), pd.DataFrame)
check(assert_type(df_str.ne("foo"), pd.DataFrame), pd.DataFrame)
check(assert_type(df_str.lt("m"), pd.DataFrame), pd.DataFrame)
check(assert_type(df_str.le("m"), pd.DataFrame), pd.DataFrame)
check(assert_type(df_str.gt("m"), pd.DataFrame), pd.DataFrame)
check(assert_type(df_str.ge("m"), pd.DataFrame), pd.DataFrame)

df_num = pd.DataFrame({"A": [1, 2], "B": [3, 4]})
check(assert_type(df_num.eq(1), pd.DataFrame), pd.DataFrame)
check(assert_type(df_num.gt(1.5), pd.DataFrame), pd.DataFrame)
check(assert_type(df_num.le(True), pd.DataFrame), pd.DataFrame)

ts = pd.Timestamp("2026-01-01")
df_ts = pd.DataFrame({"t": [ts]})
check(assert_type(df_ts.lt(ts), pd.DataFrame), pd.DataFrame)

td = pd.Timedelta(days=1)
df_td = pd.DataFrame({"d": [td]})
check(assert_type(df_td.ge(td), pd.DataFrame), pd.DataFrame)

dt = datetime.datetime(2026, 1, 1)
check(assert_type(df_ts.ne(dt), pd.DataFrame), pd.DataFrame)

# list-like and DataFrame operands remain supported
df = pd.DataFrame({"A": ["foo", "bar"], "B": [1, 2]})
check(assert_type(df.eq(["foo", 1]), pd.DataFrame), pd.DataFrame)
check(assert_type(df.eq(df), pd.DataFrame), pd.DataFrame)
check(assert_type(df.ne(pd.Series(["x", "y"])), pd.DataFrame), pd.DataFrame)

if TYPE_CHECKING_INVALID_USAGE:
_0 = df_num.eq(object()) # type: ignore[arg-type] # pyright: ignore[reportArgumentType] # pyrefly: ignore[bad-argument-type] # ty: ignore[invalid-argument-type]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move them next to test_types_df_to_df_comparison near line 212, where comparison tests already live

@cmp0xff

cmp0xff commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Hi, would you continue to work on this?

@cmp0xff

cmp0xff commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Please also merge main into the branch and ensure CI passes. There have been quite some changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DataFrame flexible comparison methods (eq/ne/le/lt/ge/gt) reject valid scalar values

2 participants