Mac环境下Jupyter Notebook启动PySpark Session遇Py4JJavaError问题求助

Hey there! I’ve run into this exact issue before on Mac, so let’s break down what’s going wrong and how to fix it.

What’s Causing the Error? #

The core problem is a version mismatch between your Java installation and Spark. That error about sun.nio.ch.DirectBuffer being inaccessible happens because:

• Older Spark versions (pre-3.0) aren’t compatible with Java 11 or newer. Java 11 introduced a module system that restricts access to internal classes like sun.nio.ch.DirectBuffer, which Spark used to rely on.
• It sounds like you installed a newer Java version (probably 11+), but your Spark installation is too old to work with it.

Solutions to Try #

Option 1: Downgrade Java to Version 8 (Most Reliable for Older Spark)

Spark 2.x and older work best with Java 8. Here’s how to set this up on Mac:

1. Install Java 8 using Homebrew:

   brew install openjdk@8
   

2. Update your JAVA_HOME environment variable to point to Java 8.
   • If you use Zsh, edit ~/.zshrc:

     echo 'export JAVA_HOME=/usr/local/opt/openjdk@8/libexec/openjdk.jdk/Contents/Home' >> ~/.zshrc
     source ~/.zshrc
     

   • If you use Bash, edit ~/.bash_profile instead:

     echo 'export JAVA_HOME=/usr/local/opt/openjdk@8/libexec/openjdk.jdk/Contents/Home' >> ~/.bash_profile
     source ~/.bash_profile
     

3. Verify the change by running:

   java -version
   

   You should see output indicating Java 8 (like openjdk version "1.8.0_XXX").

Option 2: Upgrade Spark to Version 3.0+ (If You Want to Keep Newer Java)

Spark 3.0 and later fixed this compatibility issue with Java 11+. To upgrade:

1. Install the latest Spark via Homebrew:

   brew install apache-spark
   

2. Make sure your SPARK_HOME variable is set correctly (Homebrew usually sets this automatically, but you can confirm by running echo $SPARK_HOME).
3. If you’re using PySpark in Jupyter, ensure it’s pointing to the new Spark installation. You can set this temporarily in your notebook before importing PySpark:

   import os
   os.environ['SPARK_HOME'] = '/usr/local/Cellar/apache-spark/<your-spark-version>/libexec'
   

   Replace <your-spark-version> with the actual version number (like 3.5.0).

Verify Your Fix #

Once you’ve made the change:

1. Test Spark in your terminal first by running spark-shell — if it starts without errors, you’re good to go.
2. Go back to your Jupyter Notebook and re-run your Spark Session code:

   import pyspark
   from pyspark.sql import SparkSession
   spark=SparkSession.builder.appName('test').getOrCreate()
   

内容的提问来源于stack exchange，提问作者IshN