Windows7下Jupyter Notebook使用PySpark加载CSV执行操作遇错求助

Hey there, let's dig into this PySpark issue you're facing on Windows 7 with Jupyter Notebook—both RDD actions like count() and Spark CSV reads throwing errors is definitely frustrating. Here are the most common fixes I've seen resolve this scenario:

PySpark depends entirely on Java, so this is the first check you should run:

• Make sure you have a compatible JDK installed: Spark 2.x requires Java 8, and while Spark 3.x supports Java 11, Windows 7 is limited to Java 8 (since Java 11 dropped Windows 7 support).
• Set the JAVA_HOME environment variable correctly:
  • Right-click "Computer" → Properties → Advanced System Settings → Environment Variables.
  • Add a new system variable named JAVA_HOME pointing to your JDK root folder (e.g., C:\Program Files\Java\jdk1.8.0_291).
  • Add %JAVA_HOME%\bin to your system PATH variable.
• Verify it works by opening a command prompt and running java -version and javac -version—you should see the correct Java version output without errors.

Windows has unique path-handling quirks that often trip up Spark. Here's what you need to set:

• SPARK_HOME: Create a system variable pointing to your Spark installation folder (e.g., C:\spark-2.4.8-bin-hadoop2.7), then add %SPARK_HOME%\bin to your PATH.
• HADOOP_HOME: Even if you don't use Hadoop, Spark needs this for Windows-specific utilities:
  • Download winutils.exe that matches your Spark's Hadoop version (e.g., for Spark 2.4.8 with Hadoop 2.7, get the Hadoop 2.7 winutils build).
  • Create a folder like C:\hadoop\bin, place winutils.exe inside it, then set HADOOP_HOME to C:\hadoop.
  • Add %HADOOP_HOME%\bin to your PATH.

Jupyter sometimes doesn't pick up system environment variables automatically. You can fix this two ways:

Option 1: Set Variables Directly in Your Notebook #

Add this at the top of your notebook before importing PySpark:

import os
# Update these paths to match your own installations
os.environ['JAVA_HOME'] = 'C:\\Program Files\\Java\\jdk1.8.0_291'
os.environ['SPARK_HOME'] = 'C:\\spark-2.4.8-bin-hadoop2.7'
os.environ['HADOOP_HOME'] = 'C:\\hadoop'

from pyspark.sql import SparkSession
spark = SparkSession.builder \
    .master("local[*]") \
    .appName("PySparkTest") \
    .getOrCreate()

Option 2: Create a PySpark Kernel for Jupyter #

This makes the environment persistent across notebooks:

1. Run jupyter kernelspec list in a command prompt to find your kernel directory.
2. Create a new folder named pyspark inside that directory.
3. Add a kernel.json file with this content (update paths to match your setup):

{
  "display_name": "PySpark",
  "language": "python",
  "argv": [
    "python",
    "-m",
    "ipykernel",
    "-f",
    "{connection_file}"
  ],
  "env": {
    "JAVA_HOME": "C:\\Program Files\\Java\\jdk1.8.0_291",
    "SPARK_HOME": "C:\\spark-2.4.8-bin-hadoop2.7",
    "HADOOP_HOME": "C:\\hadoop",
    "PYSPARK_PYTHON": "python",
    "PYSPARK_DRIVER_PYTHON": "jupyter",
    "PYSPARK_DRIVER_PYTHON_OPTS": "notebook"
  }
}

Windows path handling can be finicky in Python:

• Use raw strings (prefix with r) or escaped backslashes (\\) for file paths. For example, instead of ../train.csv, try r"../train.csv" or ..\\train.csv.
• Relative paths can be unreliable in Jupyter—using an absolute path like C:\\data\\train.csv is often more consistent.
• Double-check that the file exists at the specified path and your user account has read permissions for it.

If all else fails, a corrupted Spark download might be the issue:

• Download a fresh copy of Spark from the official Apache site (pick a pre-built version for Hadoop to avoid extra setup).
• Extract it to a folder without spaces or special characters (avoid C:\Program Files\Spark—use C:\spark instead).

After trying these steps, run a quick test to see if things work:

# Test RDD action
lines = spark.sparkContext.textFile(r"C:\data\sample.txt")
print(lines.count())

# Test CSV read
df = spark.read.csv(r"C:\data\train.csv", header=True, inferSchema=True)
print(df.count())

If you still get errors, sharing the exact error message would help narrow things down further—but these steps fix most common Windows 7 + PySpark + Jupyter issues I've dealt with.

内容的提问来源于stack exchange，提问作者user3631357