Windows 11下Docker运行Apache Spark容器启动失败求助

问题描述 #

我是Spark与容器技术的新手，已从Docker Hub下载apache/spark镜像，在Windows 11系统中尝试运行该容器时出现启动失败问题，现将日志附上，恳请各位提供帮助。

启动日志 #

2023-03-28 11:47:04 ++ id -u
2023-03-28 11:47:04 + myuid=185
2023-03-28 11:47:04 ++ id -g
2023-03-28 11:47:04 + mygid=0
2023-03-28 11:47:04 + set +e
2023-03-28 11:47:04 ++ getent passwd 185
2023-03-28 11:47:04 + uidentry=
2023-03-28 11:47:04 + set -e
2023-03-28 11:47:04 + '[' -z '' ']'
2023-03-28 11:47:04 + '[' -w /etc/passwd ']'
2023-03-28 11:47:04 + echo '185:x:185:0:anonymous uid:/opt/spark:/bin/false'
2023-03-28 11:47:04 + '[' -z /usr/local/openjdk-11 ']'
2023-03-28 11:47:04 + SPARK_CLASSPATH=':/opt/spark/jars/*'
2023-03-28 11:47:04 + env
2023-03-28 11:47:04 + grep SPARK_JAVA_OPT_
2023-03-28 11:47:04 + sort -t_ -k4 -n
2023-03-28 11:47:04 + sed 's/[^=]*=\(.*\)/\1/g'
2023-03-28 11:47:04 + readarray -t SPARK_EXECUTOR_JAVA_OPTS
2023-03-28 11:47:04 + '[' -n '' ']'
2023-03-28 11:47:04 + '[' -z ']'
2023-03-28 11:47:04 + '[' -z ']'
2023-03-28 11:47:04 + '[' -n '' ']'
2023-03-28 11:47:04 + '[' -z ']'
2023-03-28 11:47:04 + '[' -z ']'
2023-03-28 11:47:04 + '[' -z x ']'
2023-03-28 11:47:04 + SPARK_CLASSPATH='/opt/spark/conf::/opt/spark/jars/*'
2023-03-28 11:47:04 + case "$1" in
2023-03-28 11:47:04 + echo 'Non-spark-on-k8s command provided, proceeding in pass-through mode...'
2023-03-28 11:47:04 + CMD=("$@")
2023-03-28 11:47:04 + exec /usr/bin/tini -s --
2023-03-28 11:47:04 tini (tini version 0.19.0)
2023-03-28 11:47:04 Usage: tini [OPTIONS] PROGRAM -- [ARGS] | --version
2023-03-28 11:47:04 
2023-03-28 11:47:04 Execute a program under the supervision of a valid init process (tini)
2023-03-28 11:47:04 
2023-03-28 11:47:04 Command line options:
2023-03-28 11:47:04 
2023-03-28 11:47:04   --version: Show version and exit.
2023-03-28 11:47:04   -h: Show this help message and exit.
2023-03-28 11:47:04   -s: Register as a process subreaper (requires Linux >= 3.4).
2023-03-28 11:47:04   -p SIGNAL: Trigger SIGNAL when parent dies, e.g. "-p SIGKILL".
2023-03-28 11:47:04   -v: Generate more verbose output. Repeat up to 3 times.
2023-03-28 11:47:04   -w: Print a warning when processes are getting reaped.
2023-03-28 11:47:04   -g: Send signals to the child's process group.
2023-03-28 11:47:04   -e EXIT_CODE: Remap EXIT_CODE (from 0 to 255) to 0.
2023-03-28 11:47:04   -l: Show license and exit.
2023-03-28 11:47:04 
2023-03-28 11:47:04 Environment variables:
2023-03-28 11:47:04 
2023-03-28 11:47:04   TINI_SUBREAPER: Register as a process subreaper (requires Linux >= 3.4).
2023-03-28 11:47:04   TINI_VERBOSITY: Set the verbosity level (default: 1).
2023-03-28 11:47:04   TINI_KILL_PROCESS_GROUP: Send signals to the child's process group.
2023-03-28 11:47:04 
2023-03-28 11:47:04 Non-spark-on-k8s command provided, proceeding in pass-through mode...

问题分析 #

从日志最后几行可见，容器启动脚本执行tini -s --时未指定后续要运行的程序，导致tini输出使用帮助。核心原因是启动容器时未传入有效的Spark服务命令或交互命令。

解决方法 #

• 指定Spark服务命令启动容器：
  • 启动Spark Shell（交互式编程环境）：

    docker run -it apache/spark spark-shell
    

  • 启动Spark Master节点（暴露Web UI和服务端口）：

    docker run -d -p 7077:7077 -p 8080:8080 apache/spark spark-class org.apache.spark.deploy.master.Master
    

  • 启动Spark Worker节点（需替换<master-ip>为Master节点的实际IP）：

    docker run -d apache/spark spark-class org.apache.spark.deploy.worker.Worker spark://<master-ip>:7077
    

• 若只需进入容器交互环境，可直接指定bash命令：

  docker run -it apache/spark bash
  

• 额外检查：确保Windows 11上的Docker Desktop已正常运行，且WSL2后端配置正确（Spark镜像基于Linux，Windows环境需依赖WSL2支持容器运行）。

内容的提问来源于stack exchange，提问作者MohaMed