[AURON #1816][Correctness Testing] Spark 3.3 - Add Parquet related tests #1954
Merged
cxzl25 merged 2 commits intoapache:masterfrom Jan 26, 2026
Merged
[AURON #1816][Correctness Testing] Spark 3.3 - Add Parquet related tests #1954cxzl25 merged 2 commits intoapache:masterfrom
cxzl25 merged 2 commits intoapache:masterfrom
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request adds comprehensive Parquet-related test coverage for Auron with Spark 3.3, contributing to the correctness testing initiative (issue #1745). The PR introduces 35 new test suite classes that extend Spark's existing Parquet test suites and mix in Auron's custom testing base traits to enable native engine testing.
Changes:
- Added 35 Parquet test suites covering various aspects: IO operations, query execution, schema handling, filtering, partitioning, vectorization, encoding, compression, compatibility (Avro/Protobuf/Thrift), field IDs, and datetime rebasing
- All test suites follow the wrapper pattern: extend Spark's original test class and mix in Auron's testing trait
- Tests will run with Auron's native engine enabled to verify correctness against vanilla Spark behavior
Reviewed changes
Copilot reviewed 32 out of 32 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| AuronParquetIOSuite.scala | Tests for Parquet I/O operations with SparkQueryTestsBase |
| AuronParquetQuerySuite.scala | Tests for Parquet query execution with SparkQueryTestsBase |
| AuronParquetVectorizedSuite.scala | Tests for vectorized Parquet reading with SparkQueryTestsBase |
| AuronParquetSchemaSuite.scala | Tests for Parquet schema handling with SparkTestsSharedSessionBase |
| AuronParquetSchemaPruningSuite.scala | Tests for schema pruning optimization with SparkQueryTestsBase |
| AuronParquetSchemaInferenceSuite.scala | Tests for schema inference with SparkTestsSharedSessionBase |
| AuronParquetV1FilterSuite.scala | Tests for V1 Parquet filtering with SparkQueryTestsBase |
| AuronParquetV2FilterSuite.scala | Tests for V2 Parquet filtering with SparkQueryTestsBase |
| AuronParquetV1QuerySuite.scala | Tests for V1 Parquet queries with SparkQueryTestsBase |
| AuronParquetV2QuerySuite.scala | Tests for V2 Parquet queries with SparkQueryTestsBase |
| AuronParquetV1SchemaPruningSuite.scala | Tests for V1 schema pruning with SparkQueryTestsBase |
| AuronParquetV2SchemaPruningSuite.scala | Tests for V2 schema pruning with SparkQueryTestsBase |
| AuronParquetV1PartitionDiscoverySuite.scala | Tests for V1 partition discovery with SparkQueryTestsBase |
| AuronParquetV2PartitionDiscoverySuite.scala | Tests for V2 partition discovery with SparkQueryTestsBase |
| AuronParquetPartitionDiscoverySuite.scala | Tests for general partition discovery with SparkQueryTestsBase |
| AuronParquetFileFormatSuite.scala | Tests for Parquet file format handling with SparkQueryTestsBase |
| AuronParquetFileFormatV1Suite.scala | Tests for V1 file format with SparkQueryTestsBase |
| AuronParquetFileFormatV2Suite.scala | Tests for V2 file format with SparkQueryTestsBase |
| AuronParquetEncodingSuite.scala | Tests for Parquet encoding schemes with SparkQueryTestsBase |
| AuronParquetCompressionCodecPrecedenceSuite.scala | Tests for compression codec precedence with SparkTestsSharedSessionBase |
| AuronParquetColumnIndexSuite.scala | Tests for column index usage with SparkQueryTestsBase |
| AuronParquetFieldIdIOSuite.scala | Tests for field ID I/O operations with SparkQueryTestsBase |
| AuronParquetFieldIdSchemaSuite.scala | Tests for field ID schema handling with SparkTestsSharedSessionBase |
| AuronParquetRebaseDatetimeSuite.scala | Tests for datetime rebasing with SparkQueryTestsBase |
| AuronParquetRebaseDatetimeV1Suite.scala | Tests for V1 datetime rebasing with SparkQueryTestsBase |
| AuronParquetRebaseDatetimeV2Suite.scala | Tests for V2 datetime rebasing with SparkQueryTestsBase |
| AuronParquetAvroCompatibilitySuite.scala | Tests for Avro compatibility with SparkQueryTestsBase |
| AuronParquetProtobufCompatibilitySuite.scala | Tests for Protobuf compatibility with SparkQueryTestsBase |
| AuronParquetThriftCompatibilitySuite.scala | Tests for Thrift compatibility with SparkQueryTestsBase |
| AuronParquetCompatibilityTest.scala | General compatibility tests with SparkQueryTestsBase |
| AuronParquetInteroperabilitySuite.scala | Tests for interoperability with SparkQueryTestsBase |
| AuronParquetTest.scala | Base Parquet test utilities with SparkTestsSharedSessionBase |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
cxzl25
approved these changes
Jan 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #1816
Rationale for this change
Part of the effort to add correctness testing for Auron: #1745
What changes are included in this PR?
Adds test for Parquet which extends the ParquetIOSuite in Spark
Are there any user-facing changes?
N/A
How was this patch tested?
Ran the tests using spark-tests profile