Skip to content

[AURON #1816][Correctness Testing] Spark 3.3 - Add Parquet related tests #1954

Merged
cxzl25 merged 2 commits intoapache:masterfrom
ShreyeshArangath:parquet-tests
Jan 26, 2026
Merged

[AURON #1816][Correctness Testing] Spark 3.3 - Add Parquet related tests #1954
cxzl25 merged 2 commits intoapache:masterfrom
ShreyeshArangath:parquet-tests

Conversation

@ShreyeshArangath
Copy link
Copy Markdown
Contributor

@ShreyeshArangath ShreyeshArangath commented Jan 25, 2026

Which issue does this PR close?

Closes #1816

Rationale for this change

Part of the effort to add correctness testing for Auron: #1745

What changes are included in this PR?

Adds test for Parquet which extends the ParquetIOSuite in Spark

Are there any user-facing changes?

N/A

How was this patch tested?

Ran the tests using spark-tests profile

@ShreyeshArangath ShreyeshArangath marked this pull request as draft January 25, 2026 03:40
@ShreyeshArangath ShreyeshArangath marked this pull request as ready for review January 25, 2026 04:06
@cxzl25 cxzl25 requested a review from Copilot January 26, 2026 03:14
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request adds comprehensive Parquet-related test coverage for Auron with Spark 3.3, contributing to the correctness testing initiative (issue #1745). The PR introduces 35 new test suite classes that extend Spark's existing Parquet test suites and mix in Auron's custom testing base traits to enable native engine testing.

Changes:

  • Added 35 Parquet test suites covering various aspects: IO operations, query execution, schema handling, filtering, partitioning, vectorization, encoding, compression, compatibility (Avro/Protobuf/Thrift), field IDs, and datetime rebasing
  • All test suites follow the wrapper pattern: extend Spark's original test class and mix in Auron's testing trait
  • Tests will run with Auron's native engine enabled to verify correctness against vanilla Spark behavior

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated no comments.

Show a summary per file
File Description
AuronParquetIOSuite.scala Tests for Parquet I/O operations with SparkQueryTestsBase
AuronParquetQuerySuite.scala Tests for Parquet query execution with SparkQueryTestsBase
AuronParquetVectorizedSuite.scala Tests for vectorized Parquet reading with SparkQueryTestsBase
AuronParquetSchemaSuite.scala Tests for Parquet schema handling with SparkTestsSharedSessionBase
AuronParquetSchemaPruningSuite.scala Tests for schema pruning optimization with SparkQueryTestsBase
AuronParquetSchemaInferenceSuite.scala Tests for schema inference with SparkTestsSharedSessionBase
AuronParquetV1FilterSuite.scala Tests for V1 Parquet filtering with SparkQueryTestsBase
AuronParquetV2FilterSuite.scala Tests for V2 Parquet filtering with SparkQueryTestsBase
AuronParquetV1QuerySuite.scala Tests for V1 Parquet queries with SparkQueryTestsBase
AuronParquetV2QuerySuite.scala Tests for V2 Parquet queries with SparkQueryTestsBase
AuronParquetV1SchemaPruningSuite.scala Tests for V1 schema pruning with SparkQueryTestsBase
AuronParquetV2SchemaPruningSuite.scala Tests for V2 schema pruning with SparkQueryTestsBase
AuronParquetV1PartitionDiscoverySuite.scala Tests for V1 partition discovery with SparkQueryTestsBase
AuronParquetV2PartitionDiscoverySuite.scala Tests for V2 partition discovery with SparkQueryTestsBase
AuronParquetPartitionDiscoverySuite.scala Tests for general partition discovery with SparkQueryTestsBase
AuronParquetFileFormatSuite.scala Tests for Parquet file format handling with SparkQueryTestsBase
AuronParquetFileFormatV1Suite.scala Tests for V1 file format with SparkQueryTestsBase
AuronParquetFileFormatV2Suite.scala Tests for V2 file format with SparkQueryTestsBase
AuronParquetEncodingSuite.scala Tests for Parquet encoding schemes with SparkQueryTestsBase
AuronParquetCompressionCodecPrecedenceSuite.scala Tests for compression codec precedence with SparkTestsSharedSessionBase
AuronParquetColumnIndexSuite.scala Tests for column index usage with SparkQueryTestsBase
AuronParquetFieldIdIOSuite.scala Tests for field ID I/O operations with SparkQueryTestsBase
AuronParquetFieldIdSchemaSuite.scala Tests for field ID schema handling with SparkTestsSharedSessionBase
AuronParquetRebaseDatetimeSuite.scala Tests for datetime rebasing with SparkQueryTestsBase
AuronParquetRebaseDatetimeV1Suite.scala Tests for V1 datetime rebasing with SparkQueryTestsBase
AuronParquetRebaseDatetimeV2Suite.scala Tests for V2 datetime rebasing with SparkQueryTestsBase
AuronParquetAvroCompatibilitySuite.scala Tests for Avro compatibility with SparkQueryTestsBase
AuronParquetProtobufCompatibilitySuite.scala Tests for Protobuf compatibility with SparkQueryTestsBase
AuronParquetThriftCompatibilitySuite.scala Tests for Thrift compatibility with SparkQueryTestsBase
AuronParquetCompatibilityTest.scala General compatibility tests with SparkQueryTestsBase
AuronParquetInteroperabilitySuite.scala Tests for interoperability with SparkQueryTestsBase
AuronParquetTest.scala Base Parquet test utilities with SparkTestsSharedSessionBase

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@cxzl25 cxzl25 merged commit e6944f7 into apache:master Jan 26, 2026
94 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Correctness Testing] Spark 3.3 - Add Parquet related tests

3 participants