[AURON #1531] Enable SparkAuronConfiguration and deprecated AuronConf#1532
[AURON #1531] Enable SparkAuronConfiguration and deprecated AuronConf#1532richox merged 5 commits intoapache:masterfrom
Conversation
|
@richox @merrily01 CI is green. PTAL |
| private val tc = TaskContext.get() | ||
| private val batchSize = AuronConf.BATCH_SIZE.intConf() | ||
| private val maxBatchMemorySize = AuronConf.SUGGESTED_BATCH_MEM_SIZE.intConf() | ||
| private val sparkAuronConfig: AuronConfiguration = |
There was a problem hiding this comment.
Should this be a method on AuronAdaptor? All the invocations of the adaptor to get the configuration requests a getInstance call; we can abstract this away by adding getSparkAuronConfig() as a public method in the Adaptor, WDYT?
There was a problem hiding this comment.
Should this be a method on AuronAdaptor? All the invocations of the adaptor to get the configuration requests a
getInstancecall; we can abstract this away by addinggetSparkAuronConfig()as a public method in the Adaptor, WDYT?
@ShreyeshArangath thanks for your comment. This is an excellent suggestion. I suggest we open a separate issue to discuss this. We introduced the AuronAdaptor to abstract away specific engines like Spark, Flink, and Trino, so the API does not carry the engine name. Let's discuss this issue in #1535.
|
@Tartarus0zm, you'd better to improve the description of this pull request. Current description is too simple. |
spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeConverters.scala
Outdated
Show resolved
Hide resolved
spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeConverters.scala
Outdated
Show resolved
Hide resolved
| private val extConvertProviders = ServiceLoader.load(classOf[AuronConvertProvider]).asScala | ||
|
|
||
| private val sparkAuronConfig: AuronConfiguration = | ||
| AuronAdaptor.getInstance.getAuronConfiguration |
There was a problem hiding this comment.
Could we add getAuronConfiguration in AuronAdaptor?
There was a problem hiding this comment.
@SteNicholas The API getAuronConfiguration already exists within the AuronAdaptor. Since the AuronAdaptor use the singleton pattern, the getInstance API must be called first each time.
Let's discuss this issue in #1535.
@SteNicholas Thanks for the reminder. I have updated the description. |
6 participants
Which issue does this PR close?
Closes #1531.
Rationale for this change
We have enabled the AuronAdaptor introduced in auron-core, which serves as the bridge between the native engine and the upper-layer engine. We now need to enable SparkAuronConfiguration and deprecate AuronConf for the following reasons:
What changes are included in this PR?
Replace AuronConf in classes such as AuronConverters/NativeConverters/NativeHelper/SparkUDAFWrapperContext/SparkUDTFWrapperContext with SparkAuronConfiguration.
and add
@Deprecatedfor AuronConf.Are there any user-facing changes?
No
How was this patch tested?
CI testing verification, SparkAuronConfiguration has been test with SparkAuronConfigurationTest.