部署Spark-Submit容器时出现错误，寻求排查方案

我正在部署硕士课程项目，涉及Spark（含Worker节点）、Cassandra、Flask等组件。当前启动spark-submit容器时出现错误（错误截图：）。已尝试更换多个Spark版本，但错误信息不足，无法定位问题，请求帮助排查。

使用的docker-compose.yaml文件 #

version: "3"
services:

  zookeeper:
    container_name: zookeeper
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
    hostname: zookeeper
    networks:
      - red1

  kafka:
    container_name: kafka
    image: wurstmeister/kafka:2.12-2.3.0
    ports:
      - "9092:9092"
    depends_on:
      - zookeeper
    hostname: kafka
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "flight_delay_classification_request:1:1"
    networks:
      - red1

  mongo:
    container_name: mongo
    image: mongo:4.4.2
    ports:
      - "27017:27017"
    depends_on:
      - kafka
    hostname: mongo
    networks:
      - red1

  mongo_seed:
    image: jlmendo11/mongo_seed
    container_name: mongo_seed
    depends_on:
      - mongo
    networks:
      - red1
    environment:
      - MONGO_HOST=mongo
      - MONGO_PORT=27017
    command:
      - "mongoimport --host mongo --port 27017 -d agile_data_science -c origin_dest_distances --mode upsert --type json --file /origin_dest_distances.jsonl"
    restart: on-failure

  spark-master:
    image: bde2020/spark-master:3.3.0-hadoop3.3
    container_name: spark-master
    ports:
      - "7077:7077"
      - "9001:9001"
      - "8080:8080"
    environment:
      - SPARK_LOCAL_IP=spark-master
      - SPARK_WORKLOAD=master
    volumes:
      - ../models:/models
      - ../flight_prediction/target/scala-2.12:/target/scala-2.12
    networks:
      - red1

  spark-worker-1:
    image: bde2020/spark-worker:3.3.0-hadoop3.3
    container_name: spark-worker-1
    depends_on:
      - spark-master
    ports:
      - "8081:8081"
    environment:
      - SPARK_MASTER=spark://spark-master:7077
      - SPARK_WORKLOAD=worker
      - SPARK_LOCAL_IP=spark-worker-1
    networks:
      - red1
    volumes:
      - ../models:/models
      - ../flight_prediction/target/scala-2.12:/target/scala-2.12

  spark-worker-2:
    image: bde2020/spark-worker:3.3.0-hadoop3.3
    container_name: spark-worker-2
    depends_on:
      - spark-master
    ports:
      - "8082:8081"
    environment:
      - SPARK_MASTER=spark://spark-master:7077
      - SPARK_WORKLOAD=worker
      - SPARK_LOCAL_IP=spark-worker-2
    networks:
      - red1
    volumes:
      - ../models:/models
      - ../flight_prediction/target/scala-2.12:/target/scala-2.12

  spark-submit:
    image: bde2020/spark-submit:3.3.0-hadoop3.3
    container_name: spark-submit
    depends_on:
      - spark-master
      - spark-worker-1
      - spark-worker-2
    ports:
      - "4040:4040"
    environment:
      - SPARK_MASTER=spark://spark-master:7077
      - SPARK_WORKLOAD=submitter
      - SPARK_LOCAL_IP=spark-submit
      - CLASSPATH=/scala-2.12
    command: bash -c "sleep 15; /spark/bin/spark-submit --class \"es.upm.dit.ging.predictor.MakePrediction\" --master spark://spark-master:7077 --packages com.datastax.spark:spark-cassandra-connector_2.12:3.2.0, org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.4, com.datastax.cassandra:cassandra-driver-core:4.0.0 --jars scala-2.12/flight_prediction_2.12-0.1.jar"
   # command: bash -c "sleep 15; /spark/bin/spark-submit --class \"es.upm.dit.ging.predictor.MakePrediction\" --master spark://spark-master:7077 --packages com.datastax.spark:spark-cassandra-connector_2.12:3.2.0, org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.0, com.datastax.oss:java-driver-core_4.17.0 target/scala-2.12/flight_prediction_2.12-0.1.jar"
    networks:
      - red1
    restart: on-failure
    volumes:
      - ../flight_prediction/target/scala-2.12:/scala-2.12

  flask:
    container_name: flask
    image: jlmendo11/flask2
    ports:
      - "5000:5000"
      - "9200:9200"
    depends_on:
      - mongo_seed
      - spark-master
    hostname: flask
    networks:
      - red1
    restart: on-failure

  cassandra:
    container_name: cassandra
    image: cassandra:3
    ports:
      - "9042:9042"
    networks:
      - red1
    volumes:
      - ./cassandra/init.cql:/scripts/init.cql

  cassandra_init:
    container_name: cassandra_init
    image: nuvo/docker-cqlsh  
    depends_on: 
      - cassandra
    networks:
      - red1
    volumes:
      - ./cassandra/init.cql:/scripts/init.cql
    restart: on-failure

networks:
  red1:
    driver : bridge

使用的build.sbt文件 #

name := "flight_prediction"

version := "0.1"

scalaVersion := "2.12.10"

val sparkVersion = "3.3.4"

mainClass in Compile := Some("es.upm.dit.ging.predictor.MakePrediction")

resolvers ++= Seq(
  "apache-snapshots" at "https://repository.apache.org/snapshots/"
)

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-mllib" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion,
  "org.apache.spark" %% "spark-hive" % sparkVersion,
  "org.apache.spark" %% "spark-sql-kafka-0-10" % "3.3.4",
  "org.mongodb.spark" %% "mongo-spark-connector" % "10.1.1",
  "com.datastax.spark" %% "spark-cassandra-connector" % "3.2.0",
  "com.datastax.cassandra" % "cassandra-driver-core" % "4.0.0"
)

排查建议 #

• 统一Spark版本：Spark集群镜像（bde2020/spark-master:3.3.0-hadoop3.3）使用3.3.0，而build.sbt中指定Spark版本为3.3.4，版本不一致会引发依赖冲突。建议将集群镜像升级到3.3.4，或把build.sbt中的sparkVersion改为3.3.0。
• 修复spark-submit命令格式：--packages参数内的逗号后不能有空格，正确格式应为：

  --packages com.datastax.spark:spark-cassandra-connector_2.12:3.2.0,org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.0,com.datastax.cassandra:cassandra-driver-core:4.0.0
  

• 验证Jar包路径：确认宿主机../flight_prediction/target/scala-2.12/下存在flight_prediction_2.12-0.1.jar，且容器内通过/scala-2.12/flight_prediction_2.12-0.1.jar能正常访问。
• 查看完整容器日志：执行docker logs spark-submit获取完整错误堆栈，截图仅显示部分信息，完整日志能暴露具体异常（如类缺失、依赖冲突、连接失败等）。
• 匹配Cassandra驱动版本：Cassandra 3.x适配的驱动版本为3.x系列，建议将build.sbt中的cassandra-driver-core改为3.11.4，避免版本不兼容问题。
• 检查Spark Master连通性：在spark-submit容器内执行telnet spark-master 7077，确认能正常连接到Spark Master节点，排除网络故障。

内容的提问来源于stack exchange，提问作者Hilcar