Skip to content

[MINOR] Expose Gluten and component build information to SparkConf#11838

Merged
philo-he merged 1 commit intoapache:mainfrom
jiangjiangtian:set_build_info
Apr 7, 2026
Merged

[MINOR] Expose Gluten and component build information to SparkConf#11838
philo-he merged 1 commit intoapache:mainfrom
jiangjiangtian:set_build_info

Conversation

@jiangjiangtian
Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

Currently, Gluten build information is not injected into the Spark configuration, which causes difficulties in certain troubleshooting and tracking scenarios. For example:

  1. To find out which specific Git revision a job is running on, we have to grep the driver's syslog. Since driver logs are deleted much earlier than Spark History logs, this information becomes unavailable once the syslog is purged.
  2. We have a tracking table that stores all configs of our running jobs. Without the build metadata in the config, it's hard to tell when a job starts running on a newer version of Gluten.
    Therefore, this PR sets the Gluten build information as part of the Spark configurations. With this change, we can simply check the build revision by looking at the Spark UI:
image

@github-actions github-actions Bot added the CORE works for Gluten Core label Mar 27, 2026
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

Copy link
Copy Markdown
Member

@zhouyuan zhouyuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link
Copy Markdown
Member

@philo-he philo-he left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Thanks.

@philo-he philo-he merged commit 705026d into apache:main Apr 7, 2026
59 checks passed
@philo-he philo-he changed the title [CORE][MINOR] Expose Gluten build information in Spark configuration [MINOR] Expose Gluten and component build information to SparkConf Apr 7, 2026
malinjawi pushed a commit to malinjawi/incubator-gluten that referenced this pull request Apr 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CORE works for Gluten Core

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants