Skip to content

[AURON #1442] Fix FileNotFoundException in BuildinfoInSparkUISuite by properly configuring temp event log dir#1609

Merged
merrily01 merged 2 commits intoapache:masterfrom
yew1eb:AURON_1442_2
Nov 10, 2025
Merged

[AURON #1442] Fix FileNotFoundException in BuildinfoInSparkUISuite by properly configuring temp event log dir#1609
merrily01 merged 2 commits intoapache:masterfrom
yew1eb:AURON_1442_2

Conversation

@yew1eb
Copy link
Contributor

@yew1eb yew1eb commented Nov 6, 2025

Which issue does this PR close?

Closes #1442 .

Rationale for this change

The original beforeAll() called super.beforeAll() first, which initializes the SparkSession (and thus the SparkContext). Spark’s EventLoggingListener starts during SparkContext construction and immediately validates spark.eventLog.dir.
The subsequent line val eventLogDir = Utils.createTempDir("/tmp/spark-events") creates the directory after the listener has already failed with:

An exception or error caused a run to abort: File file:/private/tmp/spark-events does not exist 
java.io.FileNotFoundException: File file:/private/tmp/spark-events does not exist
	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:980)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:1301)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:970)
	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:462)
	at org.apache.spark.deploy.history.EventLogFileWriter.requireLogBaseDirAsDirectory(EventLogFileWriters.scala:77)
	at org.apache.spark.deploy.history.SingleEventLogFileWriter.start(EventLogFileWriters.scala:221)
	at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:83)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:622)

This PR creates a proper temporary directory before the SparkSession is initialized, configures it via sparkConf, and cleans it up afterward.

What changes are included in this PR?

Are there any user-facing changes?

No.

How was this patch tested?

Verified the test passed.

…ite by properly configuring temp event log dir
Copy link
Member

@merrily01 merrily01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @yew1eb

@merrily01 merrily01 merged commit d7e566d into apache:master Nov 10, 2025
98 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Failing unit tests as part of the BuildinfoInSparkUISuite

3 participants