NoClassDefFoundError: Could not initialize class org.apache.spark.sql.auron.NativeConverters$ during NativeShuffleExchange in Spark task

[yew1eb]

Describe the bug
The class org.apache.spark.sql.auron.NativeConverters$ fails to initialize before executing NativeShuffleExchange, causing java.lang.NoClassDefFoundError in executors. This occurs despite the class existing in the Auron JAR.

To Reproduce
Use Apache Spark 3.3 with Auron JAR (1539192)

Expected behavior

Screenshots

Additional context
Error Log:

25/11/05 10:06:49 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.sql.auron.NativeConverters$
	at org.apache.spark.sql.execution.auron.plan.NativeShuffleExchangeBase.$anonfun$nativeHashExprs$1(NativeShuffleExchangeBase.scala:92)
	at scala.collection.immutable.List.map(List.scala:293)
	at org.apache.spark.sql.execution.auron.plan.NativeShuffleExchangeBase.nativeHashExprs(NativeShuffleExchangeBase.scala:92)
	at org.apache.spark.sql.execution.auron.plan.NativeShuffleExchangeBase.$anonfun$prepareNativeShuffleDependency$8(NativeShuffleExchangeBase.scala:269)
	at org.apache.spark.sql.auron.NativePlanWrapper.plan(NativeRDD.scala:96)
	at org.apache.spark.sql.auron.NativeRDD.nativePlan(NativeRDD.scala:56)
	at org.apache.spark.sql.execution.auron.shuffle.AuronShuffleWriterBase.nativeShuffleWrite(AuronShuffleWriterBase.scala:58)
	at org.apache.spark.sql.execution.auron.plan.NativeShuffleExchangeExec$$anon$1.write(NativeShuffleExchangeExec.scala:156)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:136)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:545)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:548)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

[yew1eb]

I reproduced the failure by loading the class in a test and catch the exception stack:

try {
    Class.forName("org.apache.spark.sql.auron.NativeConverters$");
} catch (Throwable t) {
    t.printStackTrace();
}

Result:

java.lang.ExceptionInInitializerError
Caused by: java.lang.NullPointerException
    at org.apache.spark.sql.auron.NativeConverters$.<init>(NativeConverters.scala:91)
    at org.apache.spark.sql.auron.NativeConverters$.<clinit>(NativeConverters.scala)

Line 91 in NativeConverters.scala:

AuronAdaptor.getInstance.getAuronConfiguration

Root cause:
The NoClassDefFoundError is just a symptom -- the real issue is an NPE because AuronAdaptor.INSTANCE is null on Executor. Initialization of INSTANCE depends on Shims.get (a lazy val). If Shims.get is not invoked early in the Executor, AuronAdaptor.getInstance() returns null, triggering the NPE during static initialization of NativeConverters$.

Proposed fix:
Make AuronAdaptor.getInstance() self-initializing with double-checked locking and a unified default initializer:

public static AuronAdaptor getInstance() {
    if (INSTANCE == null) {
        synchronized (AuronAdaptor.class) {
            if (INSTANCE == null) {
                initializeDefault();
            }
        }
    }
    return INSTANCE;
}

public static synchronized void initializeDefault() {
    if (INSTANCE == null) {
        initInstance(new SparkAuronAdaptor());
    }
}

And update Shims.scala to use the same entry point:

object Shims {
  lazy val get: Shims = {
    AuronAdaptor.initializeDefault()
    ...
}

This ensures consistent, automatic, thread-safe initialization from both Shims.get and AuronAdaptor.getInstance(), eliminating the fragile ordering dependency.

Happy to open a PR if this approach is approved.
@Tartarus0zm @richox any feedback ?

[Tartarus0zm]

@yew1eb Great catch! Thanks for spotting this issue.
We introduced the AuronAdapter to decouple Auron from the upper-layer engine. so we can't Introduce SparkAuronAdaptor into AuronAdaptor.
It seems we still need to discuss better strategies to address this issue.
@richox Do you have any suggestions?

[Tartarus0zm]

@yew1eb How about modifying the code using the method below?

object Shims {
  AuronAdaptor.initInstance(new SparkAuronAdaptor)
  lazy val get: Shims = {
    classOf[Shims].getClassLoader
      .loadClass("org.apache.spark.sql.auron.ShimsImpl")
      .newInstance()
      .asInstanceOf[Shims]
  }
}

Would you like to test and verify it?
cc @richox

[Tartarus0zm]

It would be even better if we could add a test to verify this case.

[yew1eb]

@Tartarus0zm， thanks for the suggestion! Your trick should work in theory - since nearly all Native*Exec oeprators import org.apache.spark.sql.auron.Shims. I’ll test it in our environment and report back.

[yew1eb]

@Tartarus0zm
but unit tests like AuronCheckConvertShuffleExchangeSuite call AuronConverters.convertSparkPlan directly — which triggers AuronAdaptor.getInstance.getAuronConfiguration — without importing Shims. This means the side-effect AuronAdaptor.initInstance won’t run, leading to NPE in tests.
What's you thoughts?

[Tartarus0zm]

@Tartarus0zm but unit tests like AuronCheckConvertShuffleExchangeSuite call AuronConverters.convertSparkPlan directly — which triggers AuronAdaptor.getInstance.getAuronConfiguration — without importing Shims. This means the side-effect AuronAdaptor.initInstance won’t run, leading to NPE in tests. What's you thoughts?

@yew1eb I believe any method that can correctly reproduce this issue is acceptable, but I'm not an expert in Spark. This unit test can be reviewed by @richox .

[yew1eb]

@yew1eb How about modifying the code using the method below?

object Shims {
  AuronAdaptor.initInstance(new SparkAuronAdaptor)
  lazy val get: Shims = {
    classOf[Shims].getClassLoader
      .loadClass("org.apache.spark.sql.auron.ShimsImpl")
      .newInstance()
      .asInstanceOf[Shims]
  }
}

Would you like to test and verify it? cc @richox

@Tartarus0zm I tested your suggestion -- it doesn't work and still throws the same error.
Since the NativeShuffleExchangeExec.write execution path does not use Shims, it does not trigger class loading. As a result, the side-effect AuronAdaptor.initInstance(new SparkAuronAdaptor) never runs.
Thus, AuronAdaptor.INSTANCE remains null, and the NPE persists.
We need a more reliable initialization mechanism.

[Tartarus0zm]

@yew1eb Thanks for your feedback.
We need to carefully consider the complete solution mechanism. I have an idea: using the SPI mechanism to address this issue.
cc @richox

[yew1eb]

@Tartarus0zm
I've successfully fixed the issue using JDK SPI — it now works perfectly.
Here’s the working pseudo-code:

// auron-core
public interface AuronAdaptorProvider {
    AuronAdaptor create();
 }


public abstract class AuronAdaptor {
     public static AuronAdaptor getInstance() {
      if (INSTANCE == null) {
          synchronized (AuronAdaptor.class) {
              if (INSTANCE == null) {
                  ServiceLoader<AuronAdaptorProvider> loader = ServiceLoader.load(AuronAdaptorProvider.class);
                  for (AuronAdaptorProvider p : loader) {
                      INSTANCE = p.create();
                      break;
                  }
                  if (INSTANCE == null) {
                      throw new IllegalStateException("No AuronAdaptorProvider found");
                  }
              }
          }
        }
      }
    }
    return INSTANCE;
  }


// spark-extension
 public class SparkAuronAdaptorProvider implements AuronAdaptorProvider {
    @Override
    public AuronAdaptor create() {
        return new SparkAuronAdaptor();
    }
}

[Tartarus0zm]

@yew1eb That's wonderful!
Can you use your case to construct a test case? then verify that it has been fixed.