[AURON #1624] Support the pushdown of ORC predicates#1886
[AURON #1624] Support the pushdown of ORC predicates#1886richox merged 13 commits intoapache:masterfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR implements predicate pushdown support for ORC file reads to improve query performance by filtering data at the file level. The changes enable DataFusion predicates (WHERE clauses) to be converted to ORC predicates and pushed down to the ORC reader.
Changes:
- Added comprehensive predicate conversion logic from DataFusion physical expressions to ORC predicates
- Updated orc-rust dependency to version 17f7012 which includes predicate support
- Modified OrcExec to accept and utilize predicates during file scanning
Reviewed changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| native-engine/datafusion-ext-plans/src/orc_exec.rs | Implements predicate conversion logic supporting comparison operators, logical operators (AND/OR/NOT), IN/NOT IN, IS NULL/IS NOT NULL, and various scalar types |
| Cargo.toml | Updates orc-rust dependency to rev 17f7012 to support predicate pushdown |
| Cargo.lock | Reflects the orc-rust dependency update with new commit hash |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
update where predicate
update where predicate
update where predicate
update where predicate
update where predicate
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 3 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 3 changed files in this pull request and generated 9 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| schema: &SchemaRef, | ||
| predicates: &mut Vec<Predicate>, | ||
| ) { | ||
| if let Some(binary) = expr.as_any().downcast_ref::<BinaryExpr>() { |
There was a problem hiding this comment.
we have SCAndExpr/SCOrExpr for short-circuiting binary logic operators, could you also add a match arm for these exprs?
update where predicate
update where predicate
update where predicate
cxzl25
changed the title
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 3 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Which issue does this PR close?
Closes #1624
Rationale for this change
Support the push-down of ORC predicates to enhance the reading efficiency of ORC files
What changes are included in this PR?
Are there any user-facing changes?
no
How was this patch tested?
cluster test