Spark 2.4.5结合Scala 2.11.12与Java 21运行时遭遇InaccessibleObjectException问题求助

各位大佬好，我最近搭环境跑Spark的时候碰到了一个棘手的权限异常，折腾好一阵都没解决，特意来这儿求助！

具体情况是：我用Spark 2.4.5搭配Scala 2.11.12，Java版本是OpenJDK 21.0.6，当我尝试用sc.textFile加载一个简单文本文件时，直接抛出了InaccessibleObjectException，提示无法访问java.util.ArrayList的elementData字段，说模块java.base没有把java.util开放给未命名模块。

问题重现代码 #

我用来加载文件的代码很简单：

val lines = sc.textFile("/root/input/file1.txt")

对应的file1.txt内容如下：

Hello Spark Wordcount!
Hello everybody else!

完整报错信息 #

25/03/29 22:00:19 WARN BlockManager: Putting block broadcast_0 failed due to exception java.lang.reflect.InaccessibleObjectException: Unable to make field transient java.lang.Object[] java.util.ArrayList.elementData accessible: module java.base does not "opens java.util" to unnamed module @2462cb01.
25/03/29 22:00:19 WARN BlockManager: Block broadcast_0 could not be removed as it was not found on disk or in memory
java.lang.reflect.InaccessibleObjectException: Unable to make field transient java.lang.Object[] java.util.ArrayList.elementData accessible: module java.base does not "opens java.util" to unnamed module @2462cb01
  at java.base/java.lang.reflect.AccessibleObject.throwInaccessibleObjectException(AccessibleObject.java:391)
  at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:367)
  at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:315)
  at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:183)
  at java.base/java.lang.reflect.Field.setAccessible(Field.java:177)
  at org.apache.spark.util.SizeEstimator$$anonfun$getClassInfo$3.apply(SizeEstimator.scala:336)
  at org.apache.spark.util.SizeEstimator$$anonfun$getClassInfo$3.apply(SizeEstimator.scala:330)
  at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
  at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
  at org.apache.spark.util.SizeEstimator$.getClassInfo(SizeEstimator.scala:330)
  at org.apache.spark.util.SizeEstimator$.visitSingleObject(SizeEstimator.scala:222)
  at org.apache.spark.util.SizeEstimator$.org$apache$spark$util$SizeEstimator$$estimate(SizeEstimator.scala:201)
  at org.apache.spark.util.SizeEstimator$.estimate(SizeEstimator.scala:69)
  at org.apache.spark.util.collection.SizeTracker$class.takeSample(SizeTracker.scala:78)
  at org.apache.spark.util.collection.SizeTracker$class.afterUpdate(SizeTracker.scala:70)
  at org.apache.spark.util.collection.SizeTrackingVector.$plus$eq(SizeTrackingVector.scala:31)
  at org.apache.spark.storage.memory.DeserializedValuesHolder.storeValue(MemoryStore.scala:665)
  at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:222)
  at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
  at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
  at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
  at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
  at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)
  at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:914)
  at org.apache.spark.storage.BlockManager.putSingle(BlockManager.scala:1481)
  at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:123)
  at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:88)
  at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
  at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
  at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1489)
  at org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1035)
  at org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1027)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  at org.apache.spark.SparkContext.withScope(SparkContext.scala:699)
  at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:1027)
  at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:830)
  at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:828)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  at org.apache.spark.SparkContext.withScope(SparkContext.scala:699)
  at org.apache.spark.SparkContext.textFile(SparkContext.scala:828)
  ... 45 elided

我的环境配置 #

• Spark版本：2.4.5
• Scala版本：2.11.12
• Java版本：21.0.6 (OpenJDK 64-Bit Server VM)

已尝试的解决方法 #

我查了资料知道Java 9+的模块系统会限制反射访问，所以尝试添加了JVM参数开放权限，但没效果：

--conf "spark.driver.extraJavaOptions=--add-opens java.base/java.util=ALL-UNNAMED" --conf "spark.executor.extraJavaOptions=--add-opens java.base/java.util=ALL-UNNAMED"

有没有大佬遇到过类似问题？或者知道我哪里操作错了？麻烦指点一下，谢谢啦！

备注：内容来源于stack exchange，提问作者aymane RIhane