Fix DB-17.3 integration test failure in string_type_test [databricks]#14417
Merged
nartal1 merged 6 commits intoNVIDIA:mainfrom Mar 18, 2026
Merged
Fix DB-17.3 integration test failure in string_type_test [databricks]#14417nartal1 merged 6 commits intoNVIDIA:mainfrom
nartal1 merged 6 commits intoNVIDIA:mainfrom
Conversation
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
Collaborator
Author
|
build |
Contributor
Greptile SummaryThis PR fixes a DB 17.3 integration test failure in
Confidence Score: 4/5
Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[test_constraint_char_preserve_disabled_fallback] --> B{is_before_spark_400?}
B -- Yes --> C[SKIP: Spark 32x/33x/34x/35x not supported]
B -- No --> D{is_databricks173_or_later?}
D -- Yes DB 17.3+ --> E[assert_gpu_and_cpu_are_equal_collect\nNo StaticInvoke injected\nFull GPU execution]
D -- No Older DB or non-DB --> F[assert_gpu_fallback_collect\nStaticInvoke injected\ncpu_fallback_class_name=StaticInvoke]
G[conftest.py pytest_runtest_setup] --> H{allow_non_gpu marker present?}
H -- Yes --> I[_non_gpu_allowed = tuple from marker args]
H -- No --> J[_non_gpu_allowed = list empty]
I --> K{allow_non_gpu_conditional present?}
J --> K
K -- Yes --> L[list _non_gpu_allowed + conditional.split comma]
K -- No --> M[_non_gpu_allowed unchanged]
Last reviewed commit: 15f6152 |
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
Collaborator
Author
|
build |
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
Collaborator
Author
|
build |
gerashegalov
previously approved these changes
Mar 14, 2026
Collaborator
|
The CI failed with this error: |
Collaborator
Author
|
build |
Collaborator
Author
|
build |
jihoonson
reviewed
Mar 18, 2026
nartal1
added a commit
to nartal1/spark-rapids
that referenced
this pull request
Mar 18, 2026
…NVIDIA#14417) Fixes NVIDIA#14317 ### Description - Fixes `test_constraint_char_preserve_disabled_fallback` integration test failure on Databricks 17.3. - DB 17.3+ no longer injects `StaticInvoke` for char padding when reading Parquet with `preserveCharVarcharTypeInfo`, so the test no longer needs to assert a GPU fallback. It can run fully on GPU. - Adds `is_databricks173_or_later()` helper to `spark_session.py` and branches the test assertion accordingly. - Fix type mismatch in allow_non_gpu_conditional used without allow_non_gpu. When a test uses @allow_non_gpu_conditional without @allow_non_gpu, _non_gpu_allowed is a list (from []) but the code tried to concatenate a tuple to it. Wrap _non_gpu_allowed with list() to handle both cases. ## Test plan - [x] Ran `string_type_test.py` integration tests on DB 17.3 - all 18 passed ``` ================== 18 passed, 27 warnings in 88.57s (0:01:28) ================== ``` ### Checklists - [ ] This PR has added documentation for new or modified features or behaviors. - [x] This PR has added new tests or modified existing tests to cover new code paths. (Please explain in the PR description how the new code paths are tested, such as names of the new/existing tests that cover them.) - [ ] Performance testing has been performed and its results are added in the PR description. Or, an issue has been filed with a link in the PR description. --------- Signed-off-by: Niranjan Artal <nartal@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #14317
Description
test_constraint_char_preserve_disabled_fallbackintegration test failure on Databricks 17.3.StaticInvokefor char padding when reading Parquet withpreserveCharVarcharTypeInfo, so the test no longer needs to assert a GPU fallback. It can run fully on GPU.is_databricks173_or_later()helper tospark_session.pyand branches the test assertion accordingly.Test plan
string_type_test.pyintegration tests on DB 17.3 - all 18 passedChecklists
(Please explain in the PR description how the new code paths are tested, such as names of the new/existing tests that cover them.)