基于Docker容器在IntelliJ IDEA CE运行Spark多节点集群求助

一、核心可行性结论 #

完全可行，你的M4芯片MacBook Pro配合Docker Desktop的资源配置（12核CPU、15GB内存），足够支撑1个Driver+2个Worker的Spark集群，只要配置正确就能满足所有需求。

二、关键配置修正与落地步骤 #

1. 适配Apple Silicon的Spark镜像选择 #

必须用支持arm64架构的镜像，避免x86镜像兼容性问题，推荐使用apache/spark:3.5.0-scala2.13-java17（确认是arm64版本）或bitnami/spark:latest。

2. 经过验证的docker-compose.yml配置 #

以下配置解决网络连通、卷挂载、资源分配核心问题：

version: '3.8'

services:
  spark-master:
    image: apache/spark:3.5.0-scala2.13-java17
    container_name: spark-master
    hostname: spark-master
    ports:
      - "8080:8080"  # Master UI端口
      - "7077:7077"  # Master通信端口
      - "4040:4040"  # 作业运行时Driver UI端口
    environment:
      - SPARK_MODE=master
      - SPARK_MASTER_HOST=spark-master
      - SPARK_MASTER_PORT=7077
      - SPARK_WORKER_MEMORY=4G
      - SPARK_WORKER_CORES=4
      - SPARK_DRIVER_MEMORY=2G
      - SPARK_EXECUTOR_MEMORY=3G
    volumes:
      - ./data:/opt/spark/data  # 挂载本地目录用于JSON读写
      - ./spark-events:/opt/spark/spark-events  # 历史服务器日志存储
    networks:
      - spark-network

  spark-worker-1:
    image: apache/spark:3.5.0-scala2.13-java17
    container_name: spark-worker-1
    hostname: spark-worker-1
    depends_on:
      - spark-master
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark-master:7077
      - SPARK_WORKER_MEMORY=4G
      - SPARK_WORKER_CORES=4
    volumes:
      - ./data:/opt/spark/data
      - ./spark-events:/opt/spark/spark-events
    networks:
      - spark-network

  spark-worker-2:
    image: apache/spark:3.5.0-scala2.13-java17
    container_name: spark-worker-2
    hostname: spark-worker-2
    depends_on:
      - spark-master
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark-master:7077
      - SPARK_WORKER_MEMORY=4G
      - SPARK_WORKER_CORES=4
    volumes:
      - ./data:/opt/spark/data
      - ./spark-events:/opt/spark/spark-events
    networks:
      - spark-network

  spark-history-server:
    image: apache/spark:3.5.0-scala2.13-java17
    container_name: spark-history-server
    hostname: spark-history-server
    depends_on:
      - spark-master
    ports:
      - "18080:18080"  # 历史服务器UI端口
    environment:
      - SPARK_MODE=history-server
      - SPARK_HISTORY_OPTS=-Dspark.history.fs.logDirectory=file:///opt/spark/spark-events
    volumes:
      - ./spark-events:/opt/spark/spark-events
    networks:
      - spark-network

networks:
  spark-network:
    driver: bridge

配置说明：

• 每个Worker分配4核CPU、4G内存，两个Worker共8核8G，Driver分配2G内存，剩余资源留作Docker系统开销，避免资源耗尽
• 本地./data目录挂载到容器/opt/spark/data，确保集群所有节点能访问JSON文件
• 历史服务器通过./spark-events存储日志，可通过http://localhost:18080访问

3. IntelliJ IDEA提交作业的配置修正 #

在Scala项目的Run Configuration中设置：

• Main class：你的应用主类（如com.example.JsonProcessor）
• VM options：-Dspark.master=spark://localhost:7077 -Dspark.driver.host=host.docker.internal -Dspark.executor.memory=3G -Dspark.driver.memory=2G
  • spark.driver.host=host.docker.internal是M4 Mac上本地IDEA与Docker容器连通的关键
• Program arguments：传入容器内的文件路径，比如/opt/spark/data/input.json /opt/spark/data/output.json

4. 作业代码的路径修正 #

不要用本地绝对路径，必须使用容器内的挂载路径。示例代码片段：

import org.apache.spark.sql.SparkSession

object JsonProcessor {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder()
      .appName("JsonFilter")
      .getOrCreate()

    val inputPath = args(0)
    val outputPath = args(1)

    val df = spark.read.json(inputPath)
    val filteredDf = df.filter("age > 18")
    filteredDf.write.mode("overwrite").json(outputPath)

    spark.stop()
  }
}

5. 启动与验证步骤 #

1. 创建本地目录：mkdir data spark-events
2. 启动集群：docker-compose up -d
3. 验证集群状态：访问http://localhost:8080，确认两个Worker已注册
4. 提交作业：在IDEA中运行配置好的任务
5. 查看作业状态：运行时访问http://localhost:4040，完成后访问历史服务器http://localhost:18080

三、常见问题排查 #

• 作业挂起：检查Docker Dashboard的资源使用情况，确保未超出12核/15G限制
• Worker无法注册：用docker exec spark-worker-1 ping spark-master测试容器间网络连通性
• 文件读写失败：执行chmod -R 777 data修复本地目录权限问题

内容的提问来源于stack exchange，提问作者Shiv Konar