Spark 4.0启用AQE时执行大量查询触发IllegalThreadStateException问题

问题现象 #

• 升级至Spark 4后，在同一个长期运行的Spark实例中连续执行大量包含spark.ml或Delta Lake的查询（例如estimator.fit()）时，会确定性触发IllegalThreadStateException
• 该问题在Spark 3.x中不存在，仅在连续执行大量查询（如数千个测试）时出现，执行少量查询不会触发
• 经排查，问题疑似与Adaptive Query Execution（AQE）相关

临时规避方案 #

• 通过设置配置项 spark.sql.adaptive.enabled=false 可临时规避该问题

堆栈跟踪片段 #

java.lang.IllegalThreadStateException
at java.base/java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:892)
at java.base/java.lang.Thread.<init>(Thread.java:441)
at java.base/java.util.concurrent.Executors$DefaultThreadFactory.newThread(Executors.java:660)
at org.sparkproject.guava.util.concurrent.ThreadFactoryBuilder$1.newThread(ThreadFactoryBuilder.java:177)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:630)
at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:920)
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1353)
at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:21)
at java.base/java.util.concurrent.CompletableFuture.asyncSupplyStage(CompletableFuture.java:1782)
at java.base/java.util.concurrent.CompletableFuture.supplyAsync(CompletableFuture.java:2005)
at org.apache.spark.sql.execution.SQLExecution$.withThreadLocalCaptured(SQLExecution.scala:329)
...
at org.apache.spark.ml.Predictor.fit(Predictor.scala:115)
at org.apache.spark.ml.Predictor.fit(Predictor.scala:79)
at org.apache.spark.ml.Estimator.fit(Estimator.scala:59)

疑问 #

• 是否有其他用户遇到过该问题？
• 是否了解Spark 4中的哪些变更导致了此问题？
• 是否应该提交bug（尽管复现可能有难度）？

内容的提问来源于stack exchange，提问作者Ghislain Fourny