PySpark启动SparkContext报错：Java网关进程未发送端口号即退出

Hey there, sorry to hear you've been stuck on this for four days—total bummer when you've tried all the usual spots like StackOverflow and still can't crack it. Let's break down the common fixes for that java gateway process exited before sending the driver its port number error with your setup (Spark 2.3.0, Scala 2.12.5, Hadoop 2.7.1 on Windows):

Key Fixes to Try #

1. Fix Scala-Spark Version Mismatch #

This is probably the biggest issue here: Spark 2.3.0 does not support Scala 2.12.x. It's built specifically for Scala 2.11.x versions (the recommended one is 2.11.12).

• Downgrade Scala to 2.11.12
• Update your SCALA_HOME environment variable to point to this new installation
• Restart your Jupyter Notebook and try again

2. Verify Winutils & Hadoop Setup #

Windows relies on winutils for Hadoop filesystem operations, and small misconfigurations here often trigger gateway failures:

• Ensure HADOOP_HOME is set to your full Hadoop 2.7.1 installation folder (not just the path to winutils.exe)
• Copy winutils.exe into HADOOP_HOME/bin
• Add HADOOP_HOME/bin to your system PATH
• Create a tmp/hive folder under HADOOP_HOME, then run this command in Command Prompt to fix permissions:

  winutils.exe chmod 777 \tmp\hive
  

3. Double-Check Spark & Java Environment Variables #

• Set SPARK_HOME to your Spark 2.3.0 installation directory
• Add SPARK_HOME/bin to your system PATH
• Confirm JAVA_HOME points to JDK 8 (Spark 2.3.0 isn't fully compatible with JDK 11 or newer, which can cause initialization crashes)

4. Adjust PySpark Initialization in Jupyter #

Instead of manually creating the SparkContext, use the newer SparkSession API with findspark for smoother initialization:

import findspark
findspark.init()
from pyspark.sql import SparkSession

# Initialize SparkSession first
spark = SparkSession.builder.master("local").appName("MyTestApp").getOrCreate()
# Get SparkContext from the session
sc = spark.sparkContext

This avoids common initialization quirks that come with directly instantiating SparkContext.

5. Dig Into Hidden Error Logs #

The gateway error message is often vague—check the Spark temp logs for more details:

• Look for files named spark-*.out in C:\Users\<YourUsername>\AppData\Local\Temp
• Open the file to see specific Java errors (like missing classes or permission issues) that will point you to the exact problem

Give these steps a try one by one—start with the version mismatch, since that's the most likely culprit given your setup. Hopefully one of these gets you up and running!

内容的提问来源于stack exchange，提问作者Iriel