Mac环境下PySpark启动报错：Java gateway进程未发送端口号

问题描述 #

通过Oracle官网安装Java 8，同时用Homebrew安装了apache-spark、Python和Scala。此前用Homebrew安装Java 8时一直遇到PyJ4错误；更换为官网单独安装Java后，运行PySpark测试代码出现运行时错误。

测试代码：

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('Test Spark').getOrCreate()

sc = spark.sparkContext

报错信息：

/Users/ainaazazi/opt/anaconda3/lib/python3.9/site-packages/pyspark/bin/spark-class: line 71: /Users/ainaazazi/opt/anaconda3/bin/java: No such file or directory
/Users/ainaazazi/opt/anaconda3/lib/python3.9/site-packages/pyspark/bin/spark-class: line 96: CMD: bad array subscript
head: illegal line count -- -1
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/var/folders/01/g4x6z44s45q86gcnbs0czn9w0000gn/T/ipykernel_4609/3137907738.py in <module>
      1 from pyspark.sql import SparkSession
      2 
----> 3 spark = SparkSession.builder.appName('Test Spark').getOrCreate()
      4 
      5 sc = spark.sparkContext

~/opt/anaconda3/lib/python3.9/site-packages/pyspark/sql/session.py in getOrCreate(self)
    226                             sparkConf.set(key, value)
    227                         # This SparkContext may be an existing one.
--> 228                         sc = SparkContext.getOrCreate(sparkConf)
    229                     # Do not update `SparkConf` for existing `SparkContext`, as it's shared
    230                     # by all sessions.

~/opt/anaconda3/lib/python3.9/site-packages/pyspark/context.py in getOrCreate(cls, conf)
    390         with SparkContext._lock:
    391             if SparkContext._active_spark_context is None:
--> 392                 SparkContext(conf=conf or SparkConf())
    393             return SparkContext._active_spark_context
    394 

~/opt/anaconda3/lib/python3.9/site-packages/pyspark/context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
    142                 " is not allowed as it is a security risk."
    143 
--> 144         SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
    145         try:
    146             self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,

~/opt/anaconda3/lib/python3.9/site-packages/pyspark/context.py in _ensure_initialized(cls, instance, gateway, conf)
    337         with SparkContext._lock:
    338             if not SparkContext._gateway:
--> 339                 SparkContext._gateway = gateway or launch_gateway(conf)
    340                 SparkContext._jvm = SparkContext._gateway.jvm
    341 

~/opt/anaconda3/lib/python3.9/site-packages/pyspark/java_gateway.py in launch_gateway(conf, popen_kwargs)
    106 
    107             if not os.path.isfile(conn_info_file):
--> 108                 raise RuntimeError("Java gateway process exited before sending its port number")
    109 
    110             with open(conn_info_file, "rb") as info:

RuntimeError: Java gateway process exited before sending its port number

注：apache-spark通过Homebrew安装，Java 8通过Oracle官网安装。需要解决Mac下JAVA_HOME和PySpark相关环境变量的配置问题。

解决方案 #

1. 定位官网安装的Java 8路径 #

打开终端，执行以下命令获取Java 8的安装路径：

/usr/libexec/java_home -v 1.8

输出示例：/Library/Java/JavaVirtualMachines/jdk1.8.0_391.jdk/Contents/Home

2. 配置环境变量 #

根据你使用的Shell类型（Mac默认是zsh，对应配置文件~/.zshrc；如果是bash则对应~/.bash_profile或~/.bashrc），编辑配置文件：

步骤：

• 终端执行命令打开配置文件（以zsh为例）：

  open -e ~/.zshrc
  

• 在文件末尾添加以下内容：

  # 配置JAVA_HOME指向官网安装的Java 8
  export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)
  # 配置SPARK_HOME（Homebrew安装的Spark路径，可通过`brew info apache-spark`查看具体版本）
  export SPARK_HOME=/usr/local/Cellar/apache-spark/3.5.0/libexec
  # 将Spark命令添加到系统路径
  export PATH=$SPARK_HOME/bin:$PATH
  # 指定PySpark使用的Python解释器路径（根据你的Python安装位置调整，比如Anaconda路径）
  export PYSPARK_PYTHON=~/opt/anaconda3/bin/python3
  

  注意：SPARK_HOME中的版本号需替换为你实际安装的apache-spark版本，执行brew info apache-spark会显示类似/usr/local/Cellar/apache-spark/3.5.0的路径，加上/libexec即可。

3. 使配置生效 #

终端执行命令加载新配置：

source ~/.zshrc

如果是bash则执行source ~/.bash_profile或source ~/.bashrc。

4. 验证配置 #

• 检查JAVA_HOME是否正确：

  echo $JAVA_HOME
  

  应输出之前获取的Java 8路径。
• 测试Spark是否能正常启动：

  spark-shell
  

  若能进入Spark Scala交互环境，说明Spark配置正常。
• 运行PySpark测试代码：
  重新执行你的测试代码，若不再报错则配置完成。

内容的提问来源于stack exchange，提问作者dataNovice