Skip to content

Fix DB-17.3 integration test failure in string_type_test [databricks]#14417

Merged
nartal1 merged 6 commits intoNVIDIA:mainfrom
nartal1:db-173-IT-string_type_test
Mar 18, 2026
Merged

Fix DB-17.3 integration test failure in string_type_test [databricks]#14417
nartal1 merged 6 commits intoNVIDIA:mainfrom
nartal1:db-173-IT-string_type_test

Conversation

@nartal1
Copy link
Copy Markdown
Collaborator

@nartal1 nartal1 commented Mar 13, 2026

Fixes #14317

Description

  • Fixes test_constraint_char_preserve_disabled_fallback integration test failure on Databricks 17.3.
  • DB 17.3+ no longer injects StaticInvoke for char padding when reading Parquet with preserveCharVarcharTypeInfo, so the test no longer needs to assert a GPU fallback. It can run fully on GPU.
  • Adds is_databricks173_or_later() helper to spark_session.py and branches the test assertion accordingly.
  • Fix type mismatch in allow_non_gpu_conditional used without allow_non_gpu. When a test uses @allow_non_gpu_conditional without @allow_non_gpu, _non_gpu_allowed is a list (from []) but the code tried to concatenate a tuple to it. Wrap _non_gpu_allowed with list() to handle both cases.

Test plan

  • Ran string_type_test.py integration tests on DB 17.3 - all 18 passed
================== 18 passed, 27 warnings in 88.57s (0:01:28) ==================

Checklists

  • This PR has added documentation for new or modified features or behaviors.
  • This PR has added new tests or modified existing tests to cover new code paths.
    (Please explain in the PR description how the new code paths are tested, such as names of the new/existing tests that cover them.)
  • Performance testing has been performed and its results are added in the PR description. Or, an issue has been filed with a link in the PR description.

Signed-off-by: Niranjan Artal <nartal@nvidia.com>
@nartal1 nartal1 self-assigned this Mar 13, 2026
@nartal1 nartal1 added the test Only impacts tests label Mar 13, 2026
@nartal1 nartal1 requested a review from a team March 13, 2026 20:18
@nartal1
Copy link
Copy Markdown
Collaborator Author

nartal1 commented Mar 13, 2026

build

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Mar 13, 2026

Greptile Summary

This PR fixes a DB 17.3 integration test failure in test_constraint_char_preserve_disabled_fallback by branching test behavior: on DB 17.3+, where StaticInvoke is no longer injected for char padding during Parquet reads, the test runs fully on GPU; on earlier versions it retains the existing fallback assertion. A secondary fix corrects a TypeError in conftest.py where _non_gpu_allowed (a list when no allow_non_gpu marker is present) was incorrectly being concatenated with a tuple via the allow_non_gpu_conditional path.

  • conftest.py: Replaces _non_gpu_allowed + tuple(...) with list(_non_gpu_allowed) + ..., resolving the TypeError: can only concatenate list (not "tuple") to list that occurred when allow_non_gpu_conditional was used without allow_non_gpu.
  • spark_session.py: Adds is_databricks173_or_later() following the existing version-helper pattern.
  • string_type_test.py: Switches decorator to @allow_non_gpu_conditional(not is_databricks173_or_later(), "ProjectExec") and branches the assertion inside the test body accordingly.

Confidence Score: 4/5

  • Safe to merge — changes are targeted, well-tested, and follow existing patterns in the codebase.
  • All three changes are minimal and focused: the conftest.py fix correctly resolves a real TypeError, the new version helper follows an established pattern, and the test branching logic is sound. The PR includes a passing test run on DB 17.3. One minor note: the existing conftest.py logic unconditionally adds the conditional executor list to _non_gpu_allowed even when condition is False (pre-existing behavior, not introduced here), meaning on DB 17.3+ a surprise ProjectExec CPU fallback would not be caught by the framework — but correctness is still verified by assert_gpu_and_cpu_are_equal_collect.
  • No files require special attention.

Important Files Changed

Filename Overview
integration_tests/src/main/python/conftest.py Fixes a TypeError when allow_non_gpu_conditional is used without allow_non_gpu — wraps _non_gpu_allowed with list() before concatenating the split result, correctly handling both list (empty default) and tuple (from marker args) cases.
integration_tests/src/main/python/spark_session.py Adds is_databricks173_or_later() helper following the established pattern of all other version helpers in this file; no issues.
integration_tests/src/main/python/string_type_test.py Replaces @allow_non_gpu("ProjectExec") with @allow_non_gpu_conditional and branches between assert_gpu_and_cpu_are_equal_collect / assert_gpu_fallback_collect based on DB version; logic is correct but relies on condition being evaluated at module-import time.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[test_constraint_char_preserve_disabled_fallback] --> B{is_before_spark_400?}
    B -- Yes --> C[SKIP: Spark 32x/33x/34x/35x not supported]
    B -- No --> D{is_databricks173_or_later?}
    D -- Yes DB 17.3+ --> E[assert_gpu_and_cpu_are_equal_collect\nNo StaticInvoke injected\nFull GPU execution]
    D -- No Older DB or non-DB --> F[assert_gpu_fallback_collect\nStaticInvoke injected\ncpu_fallback_class_name=StaticInvoke]

    G[conftest.py pytest_runtest_setup] --> H{allow_non_gpu marker present?}
    H -- Yes --> I[_non_gpu_allowed = tuple from marker args]
    H -- No --> J[_non_gpu_allowed = list empty]
    I --> K{allow_non_gpu_conditional present?}
    J --> K
    K -- Yes --> L[list _non_gpu_allowed + conditional.split comma]
    K -- No --> M[_non_gpu_allowed unchanged]
Loading

Last reviewed commit: 15f6152

Signed-off-by: Niranjan Artal <nartal@nvidia.com>
@nartal1
Copy link
Copy Markdown
Collaborator Author

nartal1 commented Mar 13, 2026

build

Signed-off-by: Niranjan Artal <nartal@nvidia.com>
@nartal1
Copy link
Copy Markdown
Collaborator Author

nartal1 commented Mar 14, 2026

build

gerashegalov
gerashegalov previously approved these changes Mar 14, 2026
@jihoonson
Copy link
Copy Markdown
Collaborator

The CI failed with this error:

[2026-03-14T05:43:57.871Z] ERROR ../../../../integration_tests/src/main/python/string_type_test.py::test_constraint_char_preserve_disabled_fallback[DATAGEN_SEED=1773457709, TZ=UTC, INJECT_OOM] - TypeError: can only concatenate list (not "tuple") to list

@nartal1
Copy link
Copy Markdown
Collaborator Author

nartal1 commented Mar 16, 2026

build

@nartal1
Copy link
Copy Markdown
Collaborator Author

nartal1 commented Mar 17, 2026

build

Comment thread integration_tests/src/main/python/conftest.py
Copy link
Copy Markdown
Collaborator

@jihoonson jihoonson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nartal1 nartal1 merged commit 2294dd2 into NVIDIA:main Mar 18, 2026
47 checks passed
nartal1 added a commit to nartal1/spark-rapids that referenced this pull request Mar 18, 2026
…NVIDIA#14417)

Fixes NVIDIA#14317
### Description

- Fixes `test_constraint_char_preserve_disabled_fallback` integration
test failure on Databricks 17.3.
- DB 17.3+ no longer injects `StaticInvoke` for char padding when
reading Parquet with `preserveCharVarcharTypeInfo`, so the test no
longer needs to assert a GPU fallback. It can run fully on GPU.
- Adds `is_databricks173_or_later()` helper to `spark_session.py` and
branches the test assertion accordingly.
- Fix type mismatch in allow_non_gpu_conditional used without
allow_non_gpu. When a test uses @allow_non_gpu_conditional without
@allow_non_gpu, _non_gpu_allowed is a list (from []) but the code tried
to concatenate a tuple to it. Wrap _non_gpu_allowed with list() to
handle both cases.



## Test plan
- [x] Ran `string_type_test.py` integration tests on DB 17.3 - all 18
passed
```
================== 18 passed, 27 warnings in 88.57s (0:01:28) ==================
```

### Checklists

- [ ] This PR has added documentation for new or modified features or
behaviors.
- [x] This PR has added new tests or modified existing tests to cover
new code paths.
(Please explain in the PR description how the new code paths are tested,
such as names of the new/existing tests that cover them.)
- [ ] Performance testing has been performed and its results are added
in the PR description. Or, an issue has been filed with a link in the PR
description.

---------

Signed-off-by: Niranjan Artal <nartal@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test Only impacts tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG][DBR 17.3] Integration test failures in string_type_test.py

4 participants