Skip to content

[AURON #1473] Rename module spark-extension-shims-spark3 to spark-extension-shims-spark.#1474

Merged
turboFei merged 1 commit intoapache:masterfrom
slfan1989:auron-1473
Oct 23, 2025
Merged

[AURON #1473] Rename module spark-extension-shims-spark3 to spark-extension-shims-spark.#1474
turboFei merged 1 commit intoapache:masterfrom
slfan1989:auron-1473

Conversation

@slfan1989
Copy link
Copy Markdown
Contributor

@slfan1989 slfan1989 commented Oct 20, 2025

Which issue does this PR close?

Closes #1473.

We plan to introduce support for Spark 4. Currently, the module name spark-extension-shims-spark3 is tightly coupled with Spark 3-specific versions. To make the module naming more general and version-agnostic, we should rename it to spark-extension-shims-spark.

This change will make it easier to extend support for multiple Spark versions (e.g., Spark 3.x, Spark 4.x) without duplicating shim module definitions.

Rationale for this change

The local compilation passes.

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Auron Parent Pom 7.0.0-SNAPSHOT:
[INFO] 
[INFO] Apache Auron Parent Pom ............................ SUCCESS [  2.009 s]
[INFO] Apache Auron Proto ................................. SUCCESS [ 13.495 s]
[INFO] Apache Auron Hadoop Shim 2.12 ...................... SUCCESS [  4.903 s]
[INFO] Apache Auron Core .................................. SUCCESS [  7.310 s]
[INFO] Apache Auron Common 2.12 ........................... SUCCESS [  7.611 s]
[INFO] Apache Auron Spark Version Annotation Macros 2.12 .. SUCCESS [  6.020 s]
[INFO] Apache Auron Spark UI 2.12 ......................... SUCCESS [  4.335 s]
[INFO] Apache Auron Spark Extension 2.12 .................. SUCCESS [ 24.360 s]
[INFO] Apache Auron Spark Extension Shims Spark 3.5.7_2.12  SUCCESS [ 18.153 s]
[INFO] Apache Auron Assembly .............................. SUCCESS [  9.452 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:38 min
[INFO] Finished at: 2025-10-20T11:30:21+08:00
[INFO] ------------------------------------------------------------------------

What changes are included in this PR?

Rename spark-extension-shims-spark3 to spark-extension-shims-spark.

Are there any user-facing changes?

No.

How was this patch tested?

CI

@slfan1989
Copy link
Copy Markdown
Contributor Author

@richox Could you please help review this PR? Thank you very much!

@richox
Copy link
Copy Markdown
Contributor

richox commented Oct 22, 2025

as we are using scala-paradise for conditional compiling with different spark version now, the shim package can be further merged into spark-extension module in the future.

@slfan1989
Copy link
Copy Markdown
Contributor Author

as we are using scala-paradise for conditional compiling with different spark version now, the shim package can be further merged into spark-extension module in the future.

@richox Thank you for your explanation! This approach makes sense to me as it effectively avoids code duplication and is a good optimization.

@slfan1989
Copy link
Copy Markdown
Contributor Author

@cxzl25 @richox Could we merge this PR to proceed with the compatibility development for Spark 4? Thanks a lot.

@turboFei turboFei merged commit 327b9dd into apache:master Oct 23, 2025
85 checks passed
@slfan1989
Copy link
Copy Markdown
Contributor Author

@cxzl25 @richox @turboFei Thank you very much for reviewing the code!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rename module spark-extension-shims-spark3 to spark-extension-shims-spark

4 participants