Skip to content

[AURON #1417] Expand build support to modern Linux distributions.#1418

Merged
cxzl25 merged 7 commits intoapache:masterfrom
slfan1989:multi-distro-build-support
Oct 14, 2025
Merged

[AURON #1417] Expand build support to modern Linux distributions.#1418
cxzl25 merged 7 commits intoapache:masterfrom
slfan1989:multi-distro-build-support

Conversation

@slfan1989
Copy link
Contributor

@slfan1989 slfan1989 commented Oct 7, 2025 •
edited
Loading

Which issue does this PR close?

Closes #1417.

Rationale for this change

[AURON #1417] Expand build support to modern Linux distributions.

Since CentOS 7 has reached End of Life (EOL), it is no longer suitable for use as our standard build image.
To ensure long-term maintainability, security, and compatibility of the build environment, we have decided to migrate to actively maintained distributions.
The following images are now supported for project builds:

  • Rocky Linux 8 – a stable, community-driven successor to CentOS 8, fully binary-compatible with RHEL 8;
  • Ubuntu 24.04 LTS – provides up-to-date components and long-term support, ideal for ongoing maintenance;
  • Debian 11 (Bullseye) – offers high stability and is widely adopted in CI/CD environments, but only supports compilation starting from JDK 11.

This transition ensures continuous security updates, improved ecosystem compatibility, and a more future-proof build foundation.

What changes are included in this PR?

  • Ubuntu 24.04
build-release-1  | [WARNING] See https://maven.apache.org/plugins/maven-shade-plugin/
build-release-1  | [INFO] ------------------------------------------------------------------------
build-release-1  | [INFO] Reactor Summary for Auron Parent Pom 7.0.0-SNAPSHOT:
build-release-1  | [INFO]
build-release-1  | [INFO] Auron Parent Pom ................................... SUCCESS [  3.939 s]
build-release-1  | [INFO] proto .............................................. SUCCESS [ 24.072 s]
build-release-1  | [INFO] hadoop-shim_2.12 ................................... SUCCESS [ 10.395 s]
build-release-1  | [INFO] auron-core ......................................... SUCCESS [26:07 min]
build-release-1  | [INFO] auron-common_2.12 .................................. SUCCESS [ 10.348 s]
build-release-1  | [INFO] spark-version-annotation-macros_2.12 ............... SUCCESS [  3.209 s]
build-release-1  | [INFO] spark-extension_2.12 ............................... SUCCESS [ 33.639 s]
build-release-1  | [INFO] spark-extension-shims-spark3_2.12 .................. SUCCESS [ 27.514 s]
build-release-1  | [INFO] auron-celeborn_2.12 ................................ SUCCESS [  1.336 s]
build-release-1  | [INFO] auron-uniffle_2.12 ................................. SUCCESS [ 11.210 s]
build-release-1  | [INFO] auron-paimon_2.12 .................................. SUCCESS [  3.235 s]
build-release-1  | [INFO] assembly ........................................... SUCCESS [ 14.846 s]
build-release-1  | [INFO] ------------------------------------------------------------------------
build-release-1  | [INFO] BUILD SUCCESS
build-release-1  | [INFO] ------------------------------------------------------------------------
  • Rockylinux8
build-release-1  | [INFO] ------------------------------------------------------------------------
build-release-1  | [INFO] Reactor Summary for Auron Parent Pom 7.0.0-SNAPSHOT:
build-release-1  | [INFO]
build-release-1  | [INFO] Auron Parent Pom ................................... SUCCESS [  3.344 s]
build-release-1  | [INFO] proto .............................................. SUCCESS [ 23.813 s]
build-release-1  | [INFO] hadoop-shim_2.12 ................................... SUCCESS [ 10.253 s]
build-release-1  | [INFO] auron-core ......................................... SUCCESS [25:55 min]
build-release-1  | [INFO] auron-common_2.12 .................................. SUCCESS [ 10.243 s]
build-release-1  | [INFO] spark-version-annotation-macros_2.12 ............... SUCCESS [  2.656 s]
build-release-1  | [INFO] spark-extension_2.12 ............................... SUCCESS [ 33.988 s]
build-release-1  | [INFO] spark-extension-shims-spark3_2.12 .................. SUCCESS [ 27.955 s]
build-release-1  | [INFO] auron-celeborn_2.12 ................................ SUCCESS [  1.759 s]
build-release-1  | [INFO] auron-uniffle_2.12 ................................. SUCCESS [  2.390 s]
build-release-1  | [INFO] auron-paimon_2.12 .................................. SUCCESS [  3.720 s]
build-release-1  | [INFO] assembly ........................................... SUCCESS [ 13.842 s]
build-release-1  | [INFO] ------------------------------------------------------------------------
build-release-1  | [INFO] BUILD SUCCESS
build-release-1  | [INFO] ------------------------------------------------------------------------
build-release-1  | [INFO] Total time:  27:41 min (Wall Clock)
build-release-1  | [INFO] Finished at: 2025-10-08T09:40:57Z
build-release-1  | [INFO] ------------------------------------------------------------------------
  • Debian11

Debian 11 only provides official support starting from JDK 11, as JDK 8 dependencies have been removed from its repositories. While it’s still possible to install JDK 8 manually, given the industry’s shift toward JDK 17 and beyond, I think it’s reasonable to keep things as they are.

build-release-1  | [INFO] ------------------------------------------------------------------------
build-release-1  | [INFO] Reactor Summary for Auron Parent Pom 7.0.0-SNAPSHOT:
build-release-1  | [INFO]
build-release-1  | [INFO] Auron Parent Pom ................................... SUCCESS [  4.090 s]
build-release-1  | [INFO] proto .............................................. SUCCESS [ 26.909 s]
build-release-1  | [INFO] hadoop-shim_2.12 ................................... SUCCESS [ 10.218 s]
build-release-1  | [INFO] auron-core ......................................... SUCCESS [25:41 min]
build-release-1  | [INFO] auron-common_2.12 .................................. SUCCESS [ 10.121 s]
build-release-1  | [INFO] spark-version-annotation-macros_2.12 ............... SUCCESS [  3.417 s]
build-release-1  | [INFO] spark-extension_2.12 ............................... SUCCESS [ 34.909 s]
build-release-1  | [INFO] spark-extension-shims-spark3_2.12 .................. SUCCESS [ 30.850 s]
build-release-1  | [INFO] auron-celeborn_2.12 ................................ SUCCESS [ 10.201 s]
build-release-1  | [INFO] auron-uniffle_2.12 ................................. SUCCESS [ 11.931 s]
build-release-1  | [INFO] auron-paimon_2.12 .................................. SUCCESS [ 12.621 s]
build-release-1  | [INFO] assembly ........................................... SUCCESS [ 12.797 s]
build-release-1  | [INFO] ------------------------------------------------------------------------
build-release-1  | [INFO] BUILD SUCCESS
build-release-1  | [INFO] ------------------------------------------------------------------------
build-release-1  | [INFO] Total time:  27:44 min (Wall Clock)
build-release-1  | [INFO] Finished at: 2025-10-09T09:26:38Z
build-release-1  | [INFO] ------------------------------------------------------------------------

Are there any user-facing changes?

No.

How was this patch tested?

Manually execute the Docker build script.

@github-actions github-actions bot added the build label Oct 9, 2025
@slfan1989 slfan1989 marked this pull request as ready for review October 9, 2025 12:38
@slfan1989
Copy link
Contributor Author

slfan1989 commented Oct 9, 2025

@richox @cxzl25 Could you please help review this PR? Thanks a lot!

Since CentOS 7 has reached End of Life (EOL), we plan to introduce a new set of standard build images — Rocky Linux 8, Ubuntu 24.04 LTS, and Debian 11 (Bullseye).

These three images are our most commonly used environments and have been verified to successfully build Auron artifacts as expected.

More background details and validation results have been added to the PR description.

cxzl25
cxzl25 reviewed Oct 9, 2025
View reviewed changes
dev/docker-build/debian11/Dockerfile Outdated
# ---------------------------------------------------------------------
RUN curl https://sh.rustup.rs -sSf -o rustup-init && \
chmod +x rustup-init && \
./rustup-init -y --default-toolchain nightly-2023-08-01-x86_64-unknown-linux-gnu && \
Copy link
Contributor

@cxzl25 cxzl25 Oct 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nightly-2025-05-09

auron/rust-toolchain.toml

Line 19 in 86ac39e

channel = "nightly-2025-05-09"

Copy link
Contributor Author

@slfan1989 slfan1989 Oct 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the explanation! I will try upgrading the plugin to the 2025-05-09 version.

@merrily01
Copy link
Member

merrily01 commented Oct 11, 2025

I had raised #1252 about this earlier, though it didn’t get traction at the time.
Thanks for implementing it ~

merrily01
merrily01 reviewed Oct 11, 2025
View reviewed changes
Copy link
Member

@merrily01 merrily01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor suggestion: could you update the commit message from [AURON#1417] to [AURON #1417] (add a space) so that the issue link works correctly? @slfan1989

@merrily01 merrily01 changed the title [AURON#1417] Expand build support to modern Linux distributions. [AURON #1417] Expand build support to modern Linux distributions. Oct 13, 2025
@slfan1989
Copy link
Contributor Author

slfan1989 commented Oct 13, 2025 •
edited
Loading

  • Centos7 With nightly-2025-05-09
build-release-1  | [INFO] ------------------------------------------------------------------------
build-release-1  | [INFO] Reactor Summary for Auron Parent Pom 7.0.0-SNAPSHOT:
build-release-1  | [INFO]
build-release-1  | [INFO] Auron Parent Pom ................................... SUCCESS [  3.752 s]
build-release-1  | [INFO] proto .............................................. SUCCESS [ 24.758 s]
build-release-1  | [INFO] hadoop-shim_2.12 ................................... SUCCESS [ 10.899 s]
build-release-1  | [INFO] auron-core ......................................... SUCCESS [26:16 min]
build-release-1  | [INFO] auron-common_2.12 .................................. SUCCESS [ 10.851 s]
build-release-1  | [INFO] spark-version-annotation-macros_2.12 ............... SUCCESS [  3.018 s]
build-release-1  | [INFO] spark-extension_2.12 ............................... SUCCESS [ 36.104 s]
build-release-1  | [INFO] spark-extension-shims-spark3_2.12 .................. SUCCESS [ 34.143 s]
build-release-1  | [INFO] auron-celeborn_2.12 ................................ SUCCESS [ 12.738 s]
build-release-1  | [INFO] auron-uniffle_2.12 ................................. SUCCESS [ 13.759 s]
build-release-1  | [INFO] auron-paimon_2.12 .................................. SUCCESS [ 13.960 s]
build-release-1  | [INFO] assembly ........................................... SUCCESS [ 13.892 s]
build-release-1  | [INFO] ------------------------------------------------------------------------
build-release-1  | [INFO] BUILD SUCCESS
build-release-1  | [INFO] ------------------------------------------------------------------------
build-release-1  | [INFO] Total time:  28:24 min (Wall Clock)
build-release-1  | [INFO] Finished at: 2025-10-13T03:07:21Z
build-release-1  | [INFO] ------------------------------------------------------------------------

@slfan1989
Copy link
Contributor Author

slfan1989 commented Oct 13, 2025

@richox @cxzl25 @zuston @merrily01 Thank you all for helping with the review! We are currently testing the compatibility of CentOS 7, Ubuntu 24.04, Debian 11, and Rocky Linux 8 with the nightly-2025-05-09 build to ensure that the compilation process runs smoothly across all distributions.

@slfan1989
Copy link
Contributor Author

slfan1989 commented Oct 14, 2025 •
edited
Loading

@richox @cxzl25 @zuston @merrily01 Thank you all for helping with the review! We are currently testing the compatibility of CentOS 7, Ubuntu 24.04, Debian 11, and Rocky Linux 8 with the nightly-2025-05-09 build to ensure that the compilation process runs smoothly across all distributions.

@richox @merrily01 @cxzl25 @zuston The verification has been completed, confirming that the nightly-2025-05-09-x86_64-unknown-linux-gnu build can be successfully compiled. It has been verified to compile successfully on CentOS 7, Ubuntu 24.04, Rocky Linux 8, and Debian 11.

@slfan1989
Copy link
Contributor Author

slfan1989 commented Oct 14, 2025

Minor suggestion: could you update the commit message from [AURON#1417] to [AURON #1417] (add a space) so that the issue link works correctly? @slfan1989

Thanks for the suggestion — I’ve made the updates.

@cxzl25 cxzl25 merged commit d4e147f into apache:master Oct 14, 2025
84 checks passed
@slfan1989
Copy link
Contributor Author

slfan1989 commented Oct 14, 2025

@cxzl25 Thank you very much for helping review the code!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@merrily01 merrily01 merrily01 approved these changes

@richox richox richox approved these changes

@cxzl25 cxzl25 cxzl25 approved these changes

@zuston zuston zuston approved these changes

Assignees

No one assigned

Labels

build dev-tools

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Expand build support to modern Linux distributions

5 participants

@slfan1989 @merrily01 @richox @cxzl25 @zuston