[AURON #1638] Support scan ORC data using microsecond precision#1684
[AURON #1638] Support scan ORC data using microsecond precision#1684cxzl25 merged 5 commits intoapache:masterfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds support for configurable microsecond precision when reading ORC timestamp columns, addressing issue #1638. The implementation introduces a new boolean configuration parameter that allows users to opt into microsecond precision instead of the default nanosecond precision when scanning ORC files.
- Added configuration option
spark.auron.orc.timestamp.use.microsecondwith default valuefalse - Integrated the configuration through the JNI bridge to the Rust native execution layer
- Applied the timestamp precision setting via the orc-rust library's
ArrowReaderBuilder
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| spark-extension/src/main/java/org/apache/spark/sql/auron/AuronConf.java | Adds the ORC_TIMESTAMP_USE_MICROSECOND configuration entry with descriptive comment |
| native-engine/datafusion-ext-plans/src/orc_exec.rs | Retrieves the configuration value and conditionally applies microsecond precision to the ORC reader builder |
| native-engine/auron-jni-bridge/src/conf.rs | Defines the configuration bridge macro to expose the setting to Rust code |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
ShreyeshArangath
left a comment
There was a problem hiding this comment.
LGTM, is it possible to add a test case for this?
I tried to add it, but since Spark's UT will specify the time zone org.apache.spark.SparkFunSuite // Timezone is fixed to America/Los_Angeles for those timezone sensitive tests (timestamp_*)
TimeZone.setDefault(TimeZone.getTimeZone("America/Los_Angeles")) |
This reverts commit c5b5256.
c5b5256 to
047bbaa
Compare
Which issue does this PR close?
Closes #1638
Rationale for this change
What changes are included in this PR?
Are there any user-facing changes?
How was this patch tested?