安装指定版本PySpark时遇ExceptionInInitializerError错误求助

运行pyspark或spark-shell时出现以下错误：

Python 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.spark.unsafe.array.ByteArrayMethods.(ByteArrayMethods.java:54)
at org.apache.spark.internal.config.package$.(package.scala:825)
at org.apache.spark.internal.config.package$.(package.scala)
at org.apache.spark.deploy.SparkSubmitArguments.$anonfun$loadEnvironmentArguments$3(SparkSubmitArguments.scala:157)
at scala.Option.orElse(Option.scala:447)
at org.apache.spark.deploy.SparkSubmitArguments.loadEnvironmentArguments(SparkSubmitArguments.scala:157)
at org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:115)
at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.(SparkSubmit.scala:990)
at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:990)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:85)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module @66d18979
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
at java.base/java.lang.reflect.Constructor.checkCanSetAccessible(Constructor.java:188)
at java.base/java.lang.reflect.Constructor.setAccessible(Constructor.java:181)
at org.apache.spark.unsafe.Platform.(Platform.java:56)
... 13 more
Traceback (most recent call last):
File "C:\Apachespark\spark\python\pyspark\shell.py", line 38, in 
SparkContext._ensure_initialized()
File "C:\Apachespark\spark\python\pyspark\context.py", line 319, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "C:\Apachespark\spark\python\pyspark\java_gateway.py", line 105, in launch_gateway
raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number

已安装的指定版本软件 #

• Java version "1.8.0_261"
  Java(TM) SE Runtime Environment (build 1.8.0_261-b12)
  Java HotSpot(TM) 64-Bit Server VM (build 25.261-b12, mixed mode)
• Python 3.9.0 (tags/v3.9.0:9cf6752, Oct 5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32
• PyCharm 2022 社区版
• spark-3.0.0-preview2-bin-hadoop3.2
• winutils.exe (hadoop 3.2.0)

已完成的配置操作 #

• 在C盘创建ApacheSpark文件夹，解压Spark压缩包到该目录
• 在Spark目录内创建hadoop\bin文件夹，放入winutils.exe
• 配置系统环境变量：
  • SPARK_HOME --> C:\ApacheSpark\spark-3.0.0-preview2-bin-hadoop3.2
  • HADOOP_HOME --> C:\ApacheSpark\spark-3.0.0-preview2-bin-hadoop3.2\hadoop
• 在系统PATH中添加：
  • %SPARK_HOME%\bin
  • %HADOOP_HOME%\bin

此前将winutils.exe放在C盘单独文件夹时，会提示'pyspark' 不是内部或外部命令，也不是可运行的程序或批处理文件，调整到Spark目录下的hadoop/bin后出现上述Java网关错误。

问题原因及解决方法 #

错误核心是系统实际调用了Java 9+版本，而非你安装的Java 8。Java 8没有模块化系统，而错误中的java.base模块是Java 9及以上版本的特性，Spark 3.0.0-preview2基于Java 8开发，在高版本Java下会触发反射访问权限问题。

解决步骤 #

1. 调整PATH优先级：确保Java 8的bin目录（例如C:\Program Files\Java\jdk1.8.0_261\bin）排在所有其他Java版本的bin目录之前
2. 配置JAVA_HOME：确认JAVA_HOME环境变量指向Java 8的安装目录，而非其他版本
3. 验证Java版本：打开CMD运行java -version，确认输出为1.8.0_261，若不是则重新调整环境变量
4. 重启CMD：环境变量修改后必须重启CMD才能生效，再运行pyspark测试

额外排查点 #

• 确认winutils.exe版本严格匹配hadoop 3.2.0，避免版本不兼容
• Spark 3.0.0-preview2为预览版，若上述方法无效，可检查Python 3.9与该版本Spark的兼容性（Spark 3.0正式版支持Python 3.6-3.9，预览版理论兼容）

内容的提问来源于stack exchange，提问作者ArcReactor77