Docker环境下Spark Master与Spark Worker容器通信故障求助

问题概述 #

在Docker环境搭建Spark分布式集群时，Spark Worker容器无法连接到Spark Master，报错如下：

Caused by: java.io.IOException: Failed to connect to spark-master:7077
Caused by: java.net.UnknownHostException: spark-master

项目结构 #

repo-project/
.
├── dns
├── docker
│   ├── cloudera
│   │   ├── docker-compose.yml
│   │   └── Dockerfile
│   ├── grafana
│   │   ├── docker-compose.yml
│   │   └── Dockerfile
│   ├── mongodb
│   │   ├── docker-compose.yml
│   │   └── Dockerfile
│   ├── spark
│   │   ├── docker-compose.yml
│   │   └── Dockerfile
│   └── spark-worker
│       ├── docker-compose.yml
│       ├── Dockerfile
│       └── spark-worker.sh
├── docker-compose.yml
├── LICENSE
├── README.md
├── requirements.txt
└── src
    ├── application.py
    ├── logs
    └── Tools
        ├── __init__.py
        ├── __init__.pyc
        ├── __pycache__
        │   ├── __init__.cpython-310.pyc
        │   ├── __init__.cpython-311.pyc
        │   ├── Tools.cpython-310.pyc
        │   └── Tools.cpython-311.pyc
        └── Tools.py

已尝试方案 #

• 验证容器是否处于同一Docker网络
• 确保Spark Worker中正确配置了Spark Master的地址
• 确认Spark Master和Spark Worker容器在同一个Docker Compose中

相关配置文件 #

根目录docker-compose.yml #

version: '3'
services:
  cloudera:
    build: ./docker/cloudera
    container_name: cloudera
    hostname: quickstart.cloudera
    ports:
      - "7180:7180"
      - "8088:8088"
      - "19888:19888"
      - "50070:50070"
      - "8888:8888"
      - "8082:8080"
      - "18088:18088"
    environment:
      - CLUSTER_NAME=cluster
      - SERVICES=hdfs,hive,spark
    networks:
      - network-docker


  spark-master:
    build: ./docker/spark
    container_name: spark-master
    hostname: spark-master
    ports:
      - "8080:8080"
      - "4040:4040"
    environment:
      - SPARK_MODE=master
      - SPARK_MASTER_NAME=spark-master
      - SPARK_MASTER_UI_PORT=8080
      - SPARK_LOG_DIR=/opt/spark/logs
    depends_on:
      - cloudera
    networks:
      - network-docker

  spark-worker:
    image: bde2020/spark-worker:latest
    container_name: spark-worker
    hostname: spark-worker
    ports:
      - "8081:8081"
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER=spark://spark-master:7077
      - SPARK_WORKER_CORES=1
      - SPARK_WORKER_MEMORY=1G
      - SPARK_MASTER_UI_PORT=8080
      - SPARK_LOG_DIR=/opt/spark/logs
    depends_on:
      - spark-master
    networks:
      - network-docker

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    networks:
      - network-docker

  mongodb:
    image: mongo:latest
    container_name: mongodb
    ports:
      - "27017:27017"
    networks:
      - network-docker

networks:
  network-docker:
    driver: bridge

docker/spark/docker-compose.yml #

version: '3'
services:
  spark-master:
    build: .
    container_name: spark-master
    hostname: spark-master
    ports:
      - "8080:8080"
      - "4040:4040"
    environment:
      - SPARK_MODE=master
      - SPARK_MASTER_NAME=spark-master
      - SPARK_MASTER_UI_PORT=8080
      - SPARK_LOG_DIR=/opt/spark/logs

  spark-worker:
    image: bde2020/spark-worker:latest
    container_name: spark-worker
    hostname: spark-worker
    ports:
      - "8081:8081"
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER=spark://spark-master:7077
      - SPARK_WORKER_CORES=1
      - SPARK_WORKER_MEMORY=1G

docker/spark/Dockerfile #

FROM bde2020/spark-master:latest

COPY ./ /opt/spark-app/

# Exponha as portas necessárias
EXPOSE 8080 7077 4040

# Inicie o master do Spark
CMD ["/bin/bash", "/spark/bin/spark-class org.apache.spark.deploy.master.Master"]

解决方案 #

1. 暴露Spark Master的7077端口并明确端口配置 #

在根目录docker-compose.yml的spark-master服务中，添加7077端口映射，并明确指定Master端口环境变量：

spark-master:
  build: ./docker/spark
  container_name: spark-master
  hostname: spark-master
  ports:
    - "8080:8080"
    - "4040:4040"
    - "7077:7077"  # 新增端口映射
  environment:
    - SPARK_MODE=master
    - SPARK_MASTER_NAME=spark-master
    - SPARK_MASTER_UI_PORT=8080
    - SPARK_MASTER_PORT=7077  # 新增端口指定
    - SPARK_LOG_DIR=/opt/spark/logs
  depends_on:
    - cloudera
  networks:
    - network-docker

2. 修正Spark Master的启动命令 #

bde2020/spark-master镜像自带/master.sh启动脚本，会自动配置Master的网络参数。修改docker/spark/Dockerfile，移除自定义CMD，使用镜像默认启动逻辑：

FROM bde2020/spark-master:latest

COPY ./ /opt/spark-app/

# Exponha as portas necessárias
EXPOSE 8080 7077 4040

# 移除原CMD，使用镜像默认启动脚本

若需自定义启动，也可改为：

CMD ["/bin/bash", "/master.sh"]

3. 避免重复定义容器 #

根目录和docker/spark子目录的docker-compose.yml均定义了同名的spark-master和spark-worker容器，启动时会导致冲突。建议仅使用根目录的docker-compose.yml管理所有服务，删除子目录的docker-compose.yml或避免启动它。

4. 验证网络连通性 #

进入Spark Worker容器，执行以下命令验证是否能解析Spark Master：

docker exec -it spark-worker ping spark-master

若无法解析，重启Docker网络服务：

sudo systemctl restart docker

或重新创建自定义网络：

docker network rm network-docker
docker network create network-docker

内容的提问来源于stack exchange，提问作者Felipe