Docker环境下PySpark写入MongoDB遇数据源找不到错误求助

错误信息 #

Py4JJavaError: An error occurred while calling o48.save.
: org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND] Failed to find the data source: mongodb. Please find packages at `https://spark.apache.org/third-party-projects.html`.
    at org.apache.spark.sql.errors.QueryExecutionErrors$.dataSourceNotFoundError(QueryExecutionErrors.scala:73

环境配置 #

Dockerfile #

# Use Python 3.9 as the base image
FROM jupyter/pyspark-notebook:latest

WORKDIR /app

COPY ./requirements.txt ./

RUN conda install --yes --file requirements.txt

COPY ./src ./src

COPY ./csv_data ./csv_data

USER root

docker-compose.yml #

version: "3.7"

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: beamx
    volumes:
      - .:/app
    ports:
      - 8888:8888
    command: jupyter notebook --allow-root --ip=0.0.0.0 --port=8888 --no-browser --NotebookApp.password='sha256:1b88391210c8:20b66daeff713a802e6bd10185ffce8cf9ddb6b9c5934a4fabab218959e8612a'
    depends_on:
      - mongo

  mongo:
    image: mongo:latest
    container_name: mongo
    ports:
      - 27017:27017
    volumes:
      - ./data:/data/db

requirements.txt #

pymongo
pyspark[mongodb]

代码示例 #

from pyspark.sql import SparkSession

csv_file_path = "../csv_data/2023-07-14.csv"
# MongoDB connection details
MONGO_CONNECTION_STRING = 'mongodb://mongo:27017'
DATABASE_NAME = 'mydatabase'
COLLECTION_NAME = 'mycollection'
conn_str = MONGO_CONNECTION_STRING + '/' + DATABASE_NAME + "." + COLLECTION_NAME

conn_str

spark = SparkSession \
    .builder \
    .appName("csvTOmongo") \
    .config("spark.mongodb.read.connection.uri", "mongodb://mongo:27017/mydatabase.mycollection") \
    .config("spark.mongodb.write.connection.uri", "mongodb://mongo:27017/mydatabase.mycollection") \
    .getOrCreate()

df = spark.read.format("csv").option("header", "true").load(csv_file_path)

df.write.format("mongodb").mode("append").save()

问题原因 #

pyspark[mongodb]仅安装了Python侧的辅助依赖，但PySpark底层依赖Java版的Spark环境，连接MongoDB需要MongoDB Spark Connector（Java包）。conda安装的pyspark[mongodb]不会自动下载这个Java包，导致Spark无法识别mongodb数据源。

解决方法 #

1. 在SparkSession中指定Connector依赖 #

修改SparkSession构建代码，添加spark.jars.packages配置，指定对应版本的MongoDB Spark Connector（版本需匹配Spark和MongoDB版本）：

spark = SparkSession \
    .builder \
    .appName("csvTOmongo") \
    .config("spark.mongodb.read.connection.uri", "mongodb://mongo:27017/mydatabase.mycollection") \
    .config("spark.mongodb.write.connection.uri", "mongodb://mongo:27017/mydatabase.mycollection") \
    # 替换为匹配你Spark版本的Connector版本，示例为Spark 3.3.x + Scala 2.12
    .config("spark.jars.packages", "org.mongodb.spark:mongo-spark-connector_2.12:3.0.1") \
    .getOrCreate()

版本匹配规则：

• Spark 3.2.x → Connector 3.0.x
• Spark 3.3.x → Connector 3.0.x 或 4.0.x
• Spark 3.4.x → Connector 4.0.x
• Scala版本需与镜像中的Scala版本一致（jupyter/pyspark-notebook通常用Scala 2.12）

2. 在Dockerfile中提前安装Connector包 #

如果不想每次启动都下载依赖，可以在Dockerfile中直接下载Java包到Spark的jars目录：

# Use Python 3.9 as the base image
FROM jupyter/pyspark-notebook:latest

WORKDIR /app

COPY ./requirements.txt ./

RUN conda install --yes --file requirements.txt

# 下载MongoDB Spark Connector及依赖包到Spark jars目录
RUN wget -P $SPARK_HOME/jars/ https://repo1.maven.org/maven2/org/mongodb/spark/mongo-spark-connector_2.12/3.0.1/mongo-spark-connector_2.12-3.0.1.jar
RUN wget -P $SPARK_HOME/jars/ https://repo1.maven.org/maven2/org/mongodb/mongodb-driver-sync/4.6.1/mongodb-driver-sync-4.6.1.jar
RUN wget -P $SPARK_HOME/jars/ https://repo1.maven.org/maven2/org/mongodb/bson/4.6.1/bson-4.6.1.jar

COPY ./src ./src

COPY ./csv_data ./csv_data

USER root

3. 可选：简化requirements.txt #

pyspark[mongodb]中的mongodb后缀仅提供Python侧的辅助工具，核心依赖是Java Connector，可将requirements.txt修改为：

pyspark
pymongo  # 可选，若无需直接用pymongo操作MongoDB可移除

内容的提问来源于stack exchange，提问作者birdalugur