Skip to content

v7.0.0-incubating

Latest
Latest

Choose a tag to compare

View all tags
@merrily01 merrily01 released this 04 Mar 08:42
· 51 commits to master since this release
v7.0.0
299f00e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

New Features

  • AuronConfiguration: Introduced new configuration system with dynamic default values
  • New Modules: Added auron-core, auron-flink-extension, auron-memmgr, auron-iceberg, and auron-spark-tests modules for better modularity
  • Spark UI Integration: Introduced comprehensive Spark UI support with build info display and metrics
  • Extended Native Functions: Implemented native functions for round, pow/power, lpad, rpad, reverse, initcap, levenshtein, quarter, hour, minute, second, find_in_set, nvl, nvl2, least, greatest, isnan, bround
  • Enhanced Expressions: Native support for EqualNullSafe expression, CollectLimit, monotonically_increasing_id(), and spark_partition_id()
  • Columnar Aggregation: Implemented columnar aggregate buffers for improved performance
  • JniBridge & AuronAdaptor: Introduced for better Java-Rust interoperability

Improvements

  • Performance Optimization:
    • Improved SortMergeJoin by moving has_multi_equal out of loop
    • Optimized join operation to flush in time on duplicated keys
  • Enhanced Compatibility:
    • Support for Spark 3.5.8 and Spark 4.0.2
    • Initial compatibility support for Spark 4.1
    • Support for JDK 21
    • Upgraded from JUnit 4 to JUnit 5
  • Build & CI:
    • Unified GitHub Actions runner image to Ubuntu 24.04
    • Standardized Maven build summary output to align with Apache formatting standards
    • Introduced auron-it for enhanced CI integration testing
    • Enabled Clippy lint with -D warnings enforcement
    • Extracted Rust test logic to standalone CI workflow
  • Observability:
    • Improved Spark UI integration with build info and metrics
    • Support for DataSourceScan metrics
    • Enhanced memory logging readability
  • Configuration Management:
    • Added spark.auron.enable.broadcastexchange and spark.auron.enable.shuffleExchange configs
    • Added spark.auron.expression.singleChildFallback.enabled config
    • Introduced SparkAuronConfiguration and deprecated AuronConf
  • Dependency Updates:
    • Upgraded Spark 3.5.7 to 3.5.8
    • Bumped Uniffle from 0.9.2 to 0.10.0
    • Bumped Celeborn from 0.6.0 to 0.6.2
    • Bumped Maven from 3.9.11 to 3.9.12
  • Memory Management:
    • Introduced SparkOnHeapSpillManager for better memory control
    • Fixed possible deadlock in OnHeapSpillManager
  • Code Quality:
    • Added Maven checkstyle and scalastyle plugins
    • Removed "Exec" suffix for physical operators for cleaner naming
    • Eliminated unwrap usage and enforced Clippy lint
    • Cleaned up Clippy format/string/condition/logic lints

Bug Fixes

  • Data Correctness:
    • Fixed concat_ws null handling
    • Fixed Casting from Struct and Map to Utf8 not supported
    • Fixed NULL handling for NOT IN subquery
    • Fixed isNan semantics aligned with Spark
    • Fixed initCap semantics aligned with Spark
    • Fixed StringSplit conversion
    • Fixed ORC incorrect results from RLEv2 patched base
    • Fixed ORC check offset overflow
    • Fixed Generate index out of bounds
    • Fixed unexpected invocation of UDF
  • Type System:
    • Fixed type mismatch: cast lpad/rpad len to LongType
    • Fixed string_lower only supports literal utf8
    • Fixed Casting from Array/Map/Struct to Utf8 not supported
    • Fixed FileScanConfig.projection conversion
  • Execution Engine:
    • Fixed disable convert BroadcastExchange to native does not take effect
    • Fixed disable convert ShuffleExchange to native does not take effect
    • Fixed NativeEmptyExec should fallback on unsupported types
    • Fixed IllegalArgumentException in buildTimePartExt
    • Fixed MD5 output back to utf8
    • Fixed java.lang.NoClassDefFoundError with shaded-spark
  • Performance:
    • Fixed SortMergeJoin operation on duplicated keys
    • Fixed possible deadlock in OnHeapSpillManager
  • Ecosystem Integration:
    • Fixed data size calculation in Celeborn shuffle writer
    • Fixed UDAF fallback bug when handling DeclarativeAggregator
    • Fixed spark_normalize_nan_and_zero None
  • Testing:
    • Fixed FileNotFoundException in BuildinfoInSparkUISuite
    • Fixed NativeConvertersSuite cast trim-disable test failures
    • Fixed and enhanced nvl2 unit test
  • Documentation:
    • Clarified Docker build instructions
    • Improved README formatting and grammar
    • Added contributing guidelines

Notable Changes

  • Module Restructuring: Refactored JniBridge and AuronConf, moved to auron-core module
  • Testing Enhancement: Added comprehensive correctness testing for Spark 3.3 (Aggregate, Sort, Parquet, Functions, Expressions)
  • Docker Support: Expanded build support to modern Linux distributions and added Azure Linux 3 support
  • License Compliance: Added Apache-2.0 license declaration to all native-engine Cargo.toml files

NOTE: This release includes a significant number of new features, performance optimizations, bug fixes, and improvements across modularity, compatibility, observability, and testing. The project continues to enhance its ecosystem integration with Uniffle and Celeborn, while maintaining strong backward compatibility with Spark 3.x and forward compatibility with Spark 4.x. Some minor changes are not included in the above list, please see the commit list for more details.

What's Changed

New Contributors

Full Changelog: v6.0.0...v7.0.0

Contributors

xumingming, BrytonLee, and 18 other contributors
Assets 2
Loading