[AURON #1409] Introduce Spark UI for auron#1410
Conversation
guixiaowen
force-pushed
the
build_info
branch
3 times, most recently
from
477bc3e to
0a0a756
Compare
| <relativePath>../pom.xml</relativePath> | ||
| </parent> | ||
|
|
||
| <artifactId>auron-spark-ui_${scalaVersion}</artifactId> |
There was a problem hiding this comment.
Thanks for contributing this PR! I think we should add a name tag here — similar to what Apache Aurion Spark UI does — so that the summary information after compilation is clearer and more user-friendly.
#1414 Shows the summary information after compilation.
There was a problem hiding this comment.
Thanks for contributing this PR! I think we should add a name tag here — similar to what Apache Aurion Spark UI does — so that the summary information after compilation is clearer and more user-friendly.
#1414 Shows the summary information after compilation.
@slfan1989 Thanks, Lun, for reviewing the code. I also agree that it should be modified according to #1414. After #1414 is merged, I will make further changes based on it.
|
Because adding UI in this PR, screenshots can be provided in the description of PR. |
@cxzl25 Thank you very much for reviewing this PR. I’ve added the UI screenshots in the issue. Thanks again! |
zuston
left a comment
zuston
left a comment
There was a problem hiding this comment.
Overall lgtm. Left some minor comments
This is the good start for better observability when using the Auron
| } | ||
| content ++= | ||
| <script> | ||
| function clickDetail(details) {{ |
There was a problem hiding this comment.
do we need this? If setting the class with collapse , it looks the operation of collapse will work
There was a problem hiding this comment.
do we need this? If setting the class with
collapse, it looks the operation of collapse will work
@zuston z Thank you very much for reviewing the code. This part is indeed unnecessary, and I’ve removed it.
| <br/> | ||
| </div> | ||
|
|
||
| UIUtils.headerSparkPage(request, "Auron SQL / DataFrame", summary ++ content, parent) |
There was a problem hiding this comment.
Auron SQL / DataFrame -> Auron is enough?
There was a problem hiding this comment.
Auron SQL / DataFrame->Auronis enough?
@zuston z Thank you very much for reviewing the code. It has been updated.
| } | ||
|
|
||
| private def onSQLExecutionStart(event: SparkListenerSQLExecutionStart): Unit = { | ||
| executionIdToDescription.put(event.executionId, event.description) |
There was a problem hiding this comment.
Why we need this?
@zuston Yes, it is Yes, this is not needed here.
| with Logging { | ||
| private val executionIdToDescription = new mutable.HashMap[Long, String] | ||
|
|
||
| kvstore.addTrigger(classOf[SQLExecutionUIData], conf.get[Int](UI_RETAINED_EXECUTIONS)) { |
There was a problem hiding this comment.
Is this right? The SQLExecutionUIData should not be managed by this calss
...-spark-ui/src/main/scala/org/apache/spark/sql/execution/ui/AuronSQLHistoryServerPlugin.scala
Show resolved
Hide resolved
| extends SparkUITab(sparkUI, "auron") | ||
| with Logging { | ||
|
|
||
| override val name = "AURON SQL / DataFrame" |
There was a problem hiding this comment.
AURON is enough?
@zuston Thank you very much for reviewing the code. It has been updated.
guixiaowen
force-pushed
the
build_info
branch
from
a25a5f6 to
6cb43be
Compare
spark-extension-shims-spark3/src/main/scala/org/apache/spark/sql/auron/ShimsImpl.scala
Show resolved
Hide resolved
zuston
changed the title
|
Merged. tks @guixiaowen |
4 participants
Which issue does this PR close?
Closes #1409
Rationale for this change
Display Auron's build information on the Spark UI.
What changes are included in this PR?
Are there any user-facing changes?
Add a new spark.auron.ui.enabled parameter.
How was this patch tested?
1. the Spark application is running
When the Spark application is running, the Spark UI will display AURON SQL / DataFrame, showing Auron’s build information.
The information displayed in the Spark UI is as follows:
2. after the Spark application ends
After the Spark application ends, the information can be viewed through the Spark History Server.
You only need to package the auron-spark-ui project into a JAR and integrate it with the Spark History Server, so that the information can be viewed through the Spark History Server after the job finishes.
eg:
auron-spark-ui_2.12-7.0.0-SNAPSHOT.jar to shs jars
The information displayed in the Spark UI from Spark history server is as follows: