[AURON #1870][BUILD] Add -Xfatal-warnings to scala-maven-plugin and fix related compilation failures#1872
Conversation
0fee16e to
5241dec
Compare
1b3b198 to
7d94322
Compare
|
Depends on #1874 |
-Xfatal-warnings to scala-maven-plugin and fix related compilation failures
e62a6d8 to
5e1d6dd
Compare
There was a problem hiding this comment.
Pull request overview
This PR enables stricter Scala compilation by adding the -Xfatal-warnings flag to the scala-maven-plugin, which treats all warnings as errors. The changes also include fixes for various compilation warnings that were exposed by this new setting.
Changes:
- Added
-Xfatal-warnings,-deprecation, and-featureflags to scala-maven-plugin configuration in pom.xml - Fixed compilation warnings by adding
@nowarnannotations for temporarily unused parameters and deprecation warnings - Fixed actual code issues including unused variables, non-exhaustive pattern matches, and deprecated API usage
Reviewed changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| pom.xml | Added stricter compiler flags (-Xfatal-warnings, -deprecation, -feature) for both base config and Scala 2.13 profile |
| thirdparty/auron-uniffle/.../AuronUniffleShuffleReader.scala | Added @nowarn annotations for temporarily unused parameters |
| spark-version-annotation-macros/.../sparkver.scala | Added @nowarn annotations for macro parameters that appear unused but are used by macro expansion |
| spark-extension/.../AuronBlockStoreShuffleReaderBase.scala | Fixed UnsupportedOperationException to include error message parameter |
| spark-extension/.../NativeUnionBase.scala | Changed wildcard type parameter from _ to Any for type safety |
| spark-extension/.../NativeShuffledHashJoinBase.scala | Made pattern match exhaustive by adding catch-all case that throws IllegalArgumentException |
| spark-extension/.../NativeShuffleExchangeBase.scala | Added @nowarn annotation for temporarily unused parameters |
| spark-extension/.../NativeParquetSinkBase.scala | Replaced deprecated new Job() constructor with Job.getInstance() and added @nowarn for unused parameter |
| spark-extension/.../AuronArrowColumnVector.scala | Added @nowarn annotation for unimplemented placeholder methods |
| spark-extension/.../Shims.scala | Added @nowarn annotation for temporarily unused parameters in default implementation |
| spark-extension/.../NativeHelper.scala | Changed var to val for auronCallNativeWrapper (immutable variable) |
| spark-extension/.../NativeConverters.scala | Removed unused pattern variable names (e to _) |
| spark-extension/.../AuronConverters.scala | Added @nowarn annotations for version-specific methods with unused parameters |
| spark-extension/.../AuronCallNativeWrapper.scala | Added @nowarn annotations for deprecation warnings on JniBridge usage |
| spark-extension-shims-spark/.../AuronQuerySuite.scala | Changed unused expected variables to val _ to explicitly discard values |
| spark-extension-shims-spark/.../AuronFunctionSuite.scala | Removed completely unused variables (dateString, date) |
| spark-extension-shims-spark/.../NativeShuffledHashJoinExecProvider.scala | Added @nowarn annotations for version-specific methods with unused parameters |
| spark-extension-shims-spark/.../AuronRssShuffleManagerBase.scala | Added @nowarn annotation for temporarily unused constructor parameter |
| spark-extension-shims-spark/.../AuronBlockStoreShuffleReader.scala | Added workaround to touch unused parameter for version compatibility |
| spark-extension-shims-spark/.../ShimsImpl.scala | Added @nowarn annotations for version-specific methods with unused parameters |
| spark-extension-shims-spark/.../InterceptedValidateSparkPlan.scala | Added @nowarn annotation for version-specific method with unused parameter |
| auron-spark-ui/.../AuronSQLAppStatusListener.scala | Added @nowarn annotation for temporarily unused constructor parameter |
Comments suppressed due to low confidence (2)
spark-extension-shims-spark/src/test/scala/org/apache/auron/AuronQuerySuite.scala:217
- The computed value is assigned to '_' to explicitly discard it, but this raises the question of whether this code is needed at all. Consider removing this entire if-else expression since neither branch result is used for validation. The actual test validation appears to happen via checkSparkAnswerAndOperator on line 219.
val _ = if (forcePositionalEvolution) {
correctAnswer
} else {
Seq(Row(null, 2), Row(null, 4), Row(null, 6), Row(null, null))
}
spark-extension-shims-spark/src/test/scala/org/apache/auron/AuronQuerySuite.scala:254
- The computed value is assigned to '_' to explicitly discard it, but this raises the question of whether this code is needed at all. Consider removing this entire if-else expression since neither branch result is used for validation. The actual test validation appears to happen via checkSparkAnswerAndOperator on line 256.
val _ = if (forcePositionalEvolution) {
correctAnswer
} else {
Seq(Row(null, 2, 1), Row(null, 4, 2), Row(null, 6, 3), Row(null, null, 4))
}
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
...c/main/scala/org/apache/spark/sql/execution/auron/shuffle/AuronBlockStoreShuffleReader.scala
Show resolved
Hide resolved
a6ff2d6 to
49eb52f
Compare
7998d9e to
29af6bf
Compare
…in and fix related compilation failures
|
Rebased master, CI green. @cxzl25 PTAL |
|
cc @richox |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 24 out of 24 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (2)
spark-extension-shims-spark/src/test/scala/org/apache/auron/AuronQuerySuite.scala:217
- This
val _ = if (...) { ... } else { ... }block has no side effects and its result is ignored; it’s effectively dead code. Remove it, or (if you intended to validatecorrectAnswervs an alternative) use the computed value in an assertion/check.
val _ = if (forcePositionalEvolution) {
correctAnswer
} else {
Seq(Row(null, 2), Row(null, 4), Row(null, 6), Row(null, null))
}
spark-extension-shims-spark/src/test/scala/org/apache/auron/AuronQuerySuite.scala:254
- This
val _ = if (...) { ... } else { ... }block has no side effects and its result is ignored; it’s effectively dead code. Remove it, or (if you intended to validatecorrectAnswervs an alternative) use the computed value in an assertion/check.
val _ = if (forcePositionalEvolution) {
correctAnswer
} else {
Seq(Row(null, 2, 1), Row(null, 4, 2), Row(null, 6, 3), Row(null, null, 4))
}
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeConverters.scala
Show resolved
Hide resolved
...on/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeShuffledHashJoinBase.scala
Show resolved
Hide resolved
spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeConverters.scala
Show resolved
Hide resolved
2da9c41 to
f4d0704
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 23 out of 23 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
spark-extension-shims-spark/src/test/scala/org/apache/auron/AuronQuerySuite.scala:217
- This
val _ = if (...) { ... } else { ... }block computes an expected result but is never used, so it’s effectively dead code (kept only to silence unused warnings). Remove it, or use it in an assertion (e.g., comparespark.table("t")to the computed expected whenrequireNativeisn’t sufficient for the intent of this test).
val _ = if (forcePositionalEvolution) {
correctAnswer
} else {
Seq(Row(null, 2), Row(null, 4), Row(null, 6), Row(null, null))
}
spark-extension-shims-spark/src/test/scala/org/apache/auron/AuronQuerySuite.scala:254
- Same issue here: the computed expected rows are assigned to
_and never used, leaving dead code in the test. Remove it or make the test assert against the computed expected output.
val _ = if (forcePositionalEvolution) {
correctAnswer
} else {
Seq(Row(null, 2, 1), Row(null, 4, 2), Row(null, 6, 3), Row(null, null, 4))
}
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| sql(s"CREATE EXTERNAL TABLE t(c3 INT, c2 INT) USING ORC LOCATION '$path'") | ||
|
|
||
| val expected = if (forcePositionalEvolution) { | ||
| val _ = if (forcePositionalEvolution) { |
There was a problem hiding this comment.
It seems that this if statement is useless?
Which issue does this PR close?
Closes #1870
Rationale for this change
What changes are included in this PR?
Are there any user-facing changes?
How was this patch tested?