Spark Master Docker镜像配置SSH服务器失败问题求助

环境配置 #

docker-compose.yaml #

# this is a docker compose file for setting up an Apache Spark cluster
# 1 master and 2 worker nodes

services:
  spark-master:
    build: ./spark-master-ssh-server/
    command: bin/spark-class org.apache.spark.deploy.master.Master
    ports:
      - "9095:8080" 
      - "7777:7077"
      - "22222:22"  # exposing ssh ports
    volumes:
      - ./spark_jobs:/opt/spark_jobs
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      - spark-network

  spark-worker-1:
    build: .
    command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077
    depends_on:
      - spark-master
    environment:
      SPARK_MODE: worker
      #SPARK_WORKER_CORES: 2
      #SPARK_WORKER_MEMORY: 2g
      SPARK_MASTER_URL: spark://spark-master:7077
    volumes:
      - ./spark_jobs:/opt/spark_jobs
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      - spark-network

  spark-worker-2:
    build: .
    command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077
    depends_on:
      - spark-master
    environment:
      SPARK_MODE: worker
      #SPARK_WORKER_CORES: 2
      #SPARK_WORKER_MEMORY: 2g
      SPARK_MASTER_URL: spark://spark-master:7077
    volumes:
      - ./spark_jobs:/opt/spark_jobs
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      - spark-network


networks:
  spark-network:
    driver: bridge

Spark Master的Dockerfile（./spark-master-ssh-server/Dockerfile） #

FROM bitnami/spark:latest

# Installing SSH server
USER root
RUN apt-get update && \
 apt-get install -y openssh-server

# Creating an SSH user
RUN useradd -rm -d /home/sshuser -s /bin/bash -g root -G sudo -u 1001 sshuser

# Setting the SSH user's password
RUN echo 'sshuser:password' | chpasswd

## Generating SSH host keys
RUN ssh-keygen -A 

# Exposing the SSH port
EXPOSE 22

# Starting SSH server on container startup
CMD ["/usr/sbin/sshd", "-D"]

# Switching to sshuser
USER sshuser

问题现象 #

启动容器后进入Spark Master shell，执行service ssh status显示：

sshuser@6142e5958a8e:/opt/bitnami/spark$ service ssh status
sshd is not running ... failed!

原因分析 #

docker-compose中spark-master的command字段会覆盖Dockerfile里定义的CMD，导致容器启动时只执行了Spark Master的启动命令bin/spark-class org.apache.spark.deploy.master.Master，完全忽略了启动sshd的指令。

解决方案 #

提供两种可行方案，任选其一即可：

方案1：修改Dockerfile，同时启动sshd和Spark Master #

将Dockerfile的CMD替换为同时启动两个服务的命令，确保容器启动时两者都运行：

FROM bitnami/spark:latest

# Installing SSH server
USER root
RUN apt-get update && \
 apt-get install -y openssh-server

# Creating an SSH user
RUN useradd -rm -d /home/sshuser -s /bin/bash -g root -G sudo -u 1001 sshuser

# Setting the SSH user's password
RUN echo 'sshuser:password' | chpasswd

## Generating SSH host keys
RUN ssh-keygen -A 

# Exposing the SSH port
EXPOSE 22

# 同时启动sshd（后台运行）和Spark Master（前台运行）
CMD bash -c "/usr/sbin/sshd -D & bin/spark-class org.apache.spark.deploy.master.Master"

# Switching to sshuser
USER sshuser

修改后，删除docker-compose中spark-master的command字段，让容器使用Dockerfile的CMD启动。

方案2：修改docker-compose的command字段，同时启动两个服务 #

保留原Dockerfile，直接修改docker-compose中spark-master的command：

services:
  spark-master:
    build: ./spark-master-ssh-server/
    # 同时启动sshd和Spark Master
    command: bash -c "/usr/sbin/sshd -D & bin/spark-class org.apache.spark.deploy.master.Master"
    ports:
      - "9095:8080" 
      - "7777:7077"
      - "22222:22"  # exposing ssh ports
    volumes:
      - ./spark_jobs:/opt/spark_jobs
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      - spark-network

验证方法 #

启动容器后，进入Spark Master执行以下命令：

1. 查看进程：ps aux，应该能看到sshd和spark-class的进程
2. 检查ssh状态：service ssh status，会显示sshd正在运行
3. 外部测试连接：ssh sshuser@localhost -p 22222，输入密码password即可成功登录

内容的提问来源于stack exchange，提问作者Eya