Jupyter中Spark Session连接成功后无法创建DataFrame的问题排查

我通过以下docker-compose.yml部署了包含Hadoop、Spark及Jupyter的集群环境：

services:
  namenode:
    image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8
    container_name: namenode
    restart: always
    ports:
      - 9870:9870
      - 9000:9000
    environment:
      - CLUSTER_NAME=hadoop_cluster
      - CORE_CONF_fs_defaultFS=hdfs://namenode:9000
    volumes:
      - hadoop_namenode:/hadoop/dfs/name
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9870"]
      interval: 30s
      timeout: 10s
      retries: 3

  datanode:
    image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8
    container_name: datanode
    restart: always
    ports:
      - 9864:9864
    environment:
      - CORE_CONF_fs_defaultFS=hdfs://namenode:9000
    volumes:
      - hadoop_datanode:/hadoop/dfs/data
    depends_on:
      - namenode

  resourcemanager:
    image: bde2020/hadoop-resourcemanager:2.0.0-hadoop3.2.1-java8
    container_name: resourcemanager
    restart: always
    ports:
      - 8088:8088
    environment:
      - CORE_CONF_fs_defaultFS=hdfs://namenode:9000
      - YARN_CONF_yarn_resourcemanager_hostname=resourcemanager
    depends_on:
      - namenode

  nodemanager:
    image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8
    container_name: nodemanager
    restart: always
    ports:
      - 8042:8042
    environment:
      - CORE_CONF_fs_defaultFS=hdfs://namenode:9000
      - YARN_CONF_yarn_resourcemanager_hostname=resourcemanager
      - YARN_CONF_yarn_nodemanager_aux___services=mapreduce_shuffle
    depends_on:
      - resourcemanager

  spark-master:
    image: bitnami/spark:3.5
    container_name: spark-master
    environment:
      - SPARK_MODE=master
      - SPARK_RPC_AUTHENTICATION_ENABLED=no
      - SPARK_RPC_ENCRYPTION_ENABLED=no
      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
      - SPARK_SSL_ENABLED=no
    ports:
      - "8080:8080"
      - "7077:7077"
    volumes:
      - spark_data:/bitnami/spark

  spark-worker:
    image: bitnami/spark:3.5
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark-master:7077
      - SPARK_WORKER_MEMORY=1G
      - SPARK_WORKER_CORES=1
      - SPARK_RPC_AUTHENTICATION_ENABLED=no
      - SPARK_RPC_ENCRYPTION_ENABLED=no
      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
      - SPARK_SSL_ENABLED=no
    volumes:
      - spark_data:/bitnami/spark
    depends_on:
      - spark-master
    deploy:
      replicas: 2

  jupyter:
    image: jupyter/pyspark-notebook:latest
    container_name: jupyter
    ports:
      - "8888:8888"
    environment:
      - JUPYTER_ENABLE_LAB=yes
      - SPARK_MASTER=spark://spark-master:7077
    volumes:
      - jupyter_data:/home/jovyan/work
      - ./notebooks:/home/jovyan/work/notebooks
    depends_on:
      - spark-master

volumes:
  hadoop_namenode:
  hadoop_datanode:
  spark_data:
  jupyter_data:

在spark-shell中，先执行以下HDFS命令创建目录并设置权限：

hdfs dfs -mkdir /test/employees
hdfs dfs -chmod -R 777 /test

再通过Scala代码创建DataFrame、写入HDFS、读取返回并进行分析，操作均正常：

// First, create a dummy dataframe
val data = Seq(
  (1, "John", 30, "New York"),
  (2, "Alice", 25, "San Francisco"),
  (3, "Bob", 35, "Chicago"),
  (4, "Carol", 28, "Boston"),
  (5, "David", 40, "Seattle")
)

// Define the schema
val columns = Seq("id", "name", "age", "city")

// Create the DataFrame
val df = spark.createDataFrame(data).toDF(columns: _*)

// Show the DataFrame
df.show()

// Write to HDFS
df.write
  .mode("overwrite")
  .parquet("hdfs://namenode:9000/test/employees")

// Read back from HDFS
val dfRead = spark.read
  .parquet("hdfs://namenode:9000/test/employees")

// Show the read DataFrame
println("\nData read back from HDFS:")
dfRead.show()

// Perform some basic analysis
println("\nBasic statistics:")
dfRead.describe("age").show()

println("\nCount by city:")
dfRead.groupBy("city").count().show()

但在Jupyter Notebook中，已安装pyspark包，且Spark Session连接成功，执行以下代码时却无法创建DataFrame：

from pyspark.sql import SparkSession

# Create Spark session with proper configuration
spark = SparkSession.builder \
    .appName("JupyterTest") \
    .master("spark://spark-master:7077") \
    .config("spark.driver.host", "jupyter") \
    .config("spark.hadoop.fs.defaultFS", "hdfs://namenode:9000") \
    .getOrCreate()

# Create test DataFrame
data = [("John", 30), ("Alice", 25), ("Bob", 35)]
df = spark.createDataFrame(data, ["name", "age"])

# Show DataFrame
print("Original DataFrame:")
df.show()

# Write to HDFS
df.write.mode("overwrite").parquet("hdfs://namenode:9000/test/people")

# Read back from HDFS
df_read = spark.read.parquet("hdfs://namenode:9000/test/people")
print("\nData read from HDFS:")
df_read.show()

请问我的Spark Session配置是否存在缺失？

排查与解决方案 #

1. 核心问题：Spark版本不兼容 #

jupyter/pyspark-notebook:latest默认安装的PySpark版本可能和集群使用的Spark 3.5不匹配。PySpark版本必须与集群Spark主版本完全一致，否则会出现兼容性问题（比如无法创建DataFrame、序列化错误等）。

解决方法：
指定Jupyter镜像的Spark版本，使用jupyter/pyspark-notebook:spark-3.5替代latest，确保版本一致：

jupyter:
  image: jupyter/pyspark-notebook:spark-3.5
  # 其余配置保持不变

2. 补充必要的Spark配置 #

即使版本匹配，Jupyter容器中的PySpark可能缺少一些HDFS相关的关键配置，建议在Spark Session中补充以下配置：

spark = SparkSession.builder \
    .appName("JupyterTest") \
    .master("spark://spark-master:7077") \
    .config("spark.driver.host", "jupyter") \
    .config("spark.hadoop.fs.defaultFS", "hdfs://namenode:9000") \
    # 补充HDFS客户端配置，确保能正确解析DataNode地址
    .config("spark.hadoop.dfs.client.use.datanode.hostname", "true") \
    # 关闭权限检查（仅测试环境使用，生产环境不建议）
    .config("spark.hadoop.hadoop.security.authentication", "simple") \
    # 配置Spark序列化方式，避免跨版本序列化问题
    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
    .getOrCreate()

3. 验证HDFS目录权限 #

虽然你已经设置了/test目录为777，但可以在Jupyter中先执行HDFS命令验证连通性：

# 在Jupyter中执行HDFS命令
import subprocess
subprocess.run(["hdfs", "dfs", "-ls", "/test"], check=True)

如果执行失败，说明Jupyter容器缺少Hadoop客户端配置，需要将namenode容器中的core-site.xml和hdfs-site.xml挂载到Jupyter容器的Spark配置目录：
修改docker-compose中jupyter服务的volumes：

jupyter:
  # 其余配置不变
  volumes:
    - jupyter_data:/home/jovyan/work
    - ./notebooks:/home/jovyan/work/notebooks
    # 挂载Hadoop配置文件到Spark的conf目录
    - ./hadoop-conf:/usr/local/spark/conf

其中./hadoop-conf目录需要从namenode容器中复制配置文件：

# 从namenode容器复制Hadoop配置
docker cp namenode:/hadoop/etc/hadoop/core-site.xml ./hadoop-conf/
docker cp namenode:/hadoop/etc/hadoop/hdfs-site.xml ./hadoop-conf/

4. 检查Spark Session日志 #

如果问题仍存在，可以查看Spark Session的日志来定位具体错误：

# 在Jupyter中查看日志级别
spark.sparkContext.setLogLevel("DEBUG")
# 重新执行创建DataFrame的代码，查看输出日志中的错误信息

内容的提问来源于stack exchange，提问作者Gesang Wibawono