PySpark本地模式textFile读取文件报Py4JJavaError求助

环境信息 #

• Spark版本：spark-2.4.6-bin-hadoop2.7
• Java版本：19.0.1
• Scala版本：2.13.10
• Hadoop版本：3.0.0

测试代码 #

from pyspark import SparkConf,SparkContext

if __name__ == '__main__':
    conf=SparkConf().setMaster("local[*]").setAppName('WordCount')
    sc=SparkContext(conf=conf)
    rdd_init=sc.textFile("file:///D:/PythonCode/pythonProject/letters.txt")

已尝试的路径写法 #

• rdd_init=sc.textFile("file:///D:/PythonCode/pythonProject/letters.txt")
• rdd_init=sc.textFile("file://D:/PythonCode/pythonProject/letters.txt")
• rdd_init=sc.textFile("file:///letters.txt")
• rdd_init=sc.textFile("letters.txt")
• rdd_init=sc.textFile("file://./letters.txt")
• rdd_init=sc.textFile("file:///D:/PythonCode/pythonProject/letters.csv")

报错信息 #

py4j.protocol.Py4JJavaError: An error occurred while calling o12.textFile

详细报错栈：

Traceback (most recent call last):
  File "D:/PythonCode/pythonProject/main.py", line 8, in <module>
    rdd_init=sc.textFile("file:///D:/PythonCode/pythonProject/letters.txt")
  File "D:\PythonCode\pythonProject\venv\lib\site-packages\pyspark\context.py", line 599, in textFile
    return RDD(self._jsc.textFile(name, minPartitions), self,
  File "D:\ProgramFiles\Spark\spark-2.4.6-bin-hadoop2.7\python\lib\py4j-0.10.7-src.zip\py4j\java_gateway.py", line 1257, in __call__
  File "D:\ProgramFiles\Spark\spark-2.4.6-bin-hadoop2.7\python\lib\py4j-0.10.7-src.zip\py4j\protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o12.textFile.
: java.lang.reflect.InaccessibleObjectException: Unable to make field transient java.lang.Object[] java.util.ArrayList.elementData accessible: module java.base does not "opens java.util" to unnamed module @776a6d9b
    at java.base/java.lang.reflect.AccessibleObject.throwInaccessibleObjectException(AccessibleObject.java:387)
    at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:363)
    at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:311)
    at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:180)
    at java.base/java.lang.reflect.Field.setAccessible(Field.java:174)
    at org.apache.spark.util.SizeEstimator$$anonfun$getClassInfo$3.apply(SizeEstimator.scala:336)
    at org.apache.spark.util.SizeEstimator$$anonfun$getClassInfo$3.apply(SizeEstimator.scala:330)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
    at org.apache.spark.util.SizeEstimator$.getClassInfo(SizeEstimator.scala:330)
    at org.apache.spark.util.SizeEstimator$.visitSingleObject(SizeEstimator.scala:222)
    at org.apache.spark.util.SizeEstimator$.org$apache$spark$util$SizeEstimator$$estimate(SizeEstimator.scala:201)
    at org.apache.spark.util.SizeEstimator$.estimate(SizeEstimator.scala:69)
    at org.apache.spark.util.collection.SizeTracker$class.takeSample(SizeTracker.scala:78)
    at org.apache.spark.util.collection.SizeTracker$class.afterUpdate(SizeTracker.scala:70)
    at org.apache.spark.util.collection.SizeTrackingVector.$plus$eq(SizeTrackingVector.scala:31)
    at org.apache.spark.storage.memory.DeserializedValuesHolder.storeValue(MemoryStore.scala:665)
    at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:222)
    at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
    at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
    at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)
    at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:914)
    at org.apache.spark.storage.BlockManager.putSingle(BlockManager.scala:1481)
    at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:123)
    at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:88)
    at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
    at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
    at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1489)
    at org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1035)
    at org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1027)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.SparkContext.withScope(SparkContext.scala:699)
    at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:1027)
    at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:830)
    at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:828)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.SparkContext.withScope(SparkContext.scala:699)
    at org.apache.spark.SparkContext.textFile(SparkContext.scala:828)
    at org.apache.spark.api.java.JavaSparkContext.textFile(JavaSparkContext.scala:179)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.base/java.lang.reflect.Method.invoke(Method.java:578)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.base/java.lang.Thread.run(Thread.java:1589)

问题原因 #

Spark 2.4.6属于旧版本，不兼容Java 19。Java 9及以后引入的模块化系统，会限制Spark 2.4.x代码中使用的反射操作，触发InaccessibleObjectException，因为模块未开放对应包给未命名模块。同时，Spark 2.4.6官方适配的Scala版本为2.11或2.12，当前使用的Scala 2.13.10也存在版本不匹配问题。

解决方案 #

方案1：降级Java版本 #

Spark 2.4.6官方推荐使用Java 8，操作步骤：

1. 下载并安装JDK 1.8版本
2. 修改系统环境变量：
   • 将JAVA_HOME指向Java 8的安装目录
   • 调整Path变量，把Java 8的bin目录移到最顶端
3. 验证：打开命令行输入java -version，确认显示Java 8版本信息

方案2：添加JVM启动参数（临时解决） #

若暂时不想降级Java，可在启动PySpark时添加JVM参数开放模块权限：
修改代码添加配置：

from pyspark import SparkConf,SparkContext

if __name__ == '__main__':
    conf=SparkConf() \
        .setMaster("local[*]") \
        .setAppName('WordCount') \
        .set("spark.driver.extraJavaOptions", "--add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED")
    sc=SparkContext(conf=conf)
    rdd_init=sc.textFile("file:///D:/PythonCode/pythonProject/letters.txt")
    # 示例后续操作
    result = rdd_init.flatMap(lambda x: x.split()).map(lambda x: (x,1)).reduceByKey(lambda a,b:a+b).collect()
    print(result)
    sc.stop()

也可在系统环境变量中设置SPARK_SUBMIT_OPTS：

SPARK_SUBMIT_OPTS="--add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED"

方案3：升级Spark版本 #

升级到支持Java 19的Spark版本（如Spark 3.3及以上），同时匹配对应Scala版本（Spark 3.3适配Scala 2.12或2.13），从根源解决版本兼容问题。

额外验证路径正确性 #

确认文件路径格式：

• 本地文件使用file:///D:/PythonCode/pythonProject/letters.txt是正确格式（注意三个斜杠）
• Windows系统下也可直接使用绝对路径D:/PythonCode/pythonProject/letters.txt

内容的提问来源于stack exchange，提问作者te.he