Dockerfile构建Airflow+Spark镜像报错(exit code127)及Python依赖咨询

问题描述 #

我希望通过Dockerfile构建包含Airflow和Spark的镜像，创建可运行Airflow流水线的容器，该流水线会启动Spark任务，从GCP Bucket读取大量CSV文件并处理。编写的Dockerfile如下：

FROM apache/airflow:2.5.0

ARG SPARK_VERSION="3.1.2"
ARG HADOOP_VERSION="3.2"

# setup spark
ENV SPARK_HOME /usr/local/spark
RUN cd "/tmp" && \
    wget --no-verbose "https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" && \
    tar -xvzf "spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" && \
    mkdir -p "${SPARK_HOME}/bin" && \
    mkdir -p "${SPARK_HOME}/assembly/target/scala-2.12/jars" && \
    cp -a "spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}/bin/." "${SPARK_HOME}/bin/" && \
    cp -a "spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}/jars/." "${SPARK_HOME}/assembly/target/scala-2.12/jars/" && \
    rm "spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz"

构建镜像时出现exit code 127错误，错误信息：

docker build -t my_airflow_image .
[+] Building 0.3s (5/5) FINISHED                                                                                                                                                                           
 => [internal] load build definition from Dockerfile                                                                                                                                                   0.0s
 => => transferring dockerfile: 795B                                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/apache/airflow:2.5.0                                                                                                                                        0.0s
 => CACHED [1/2] FROM docker.io/apache/airflow:2.5.0                                                                                                                                                   0.0s
 => ERROR [2/2] RUN cd "/tmp" &&     wget --no-verbose "https://archive.apache.org/dist/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz" &&     tar -xvzf "spark-3.1.2-bin-hadoop3.2.tgz" &&     mkdi  0.2s
------
 > [2/2] RUN cd "/tmp" &&     wget --no-verbose "https://archive.apache.org/dist/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz" &&     tar -xvzf "spark-3.1.2-bin-hadoop3.2.tgz" &&     mkdir -p "/usr/local/spark/bin" &&     mkdir -p "/usr/local/spark/assembly/target/scala-2.12/jars" &&     cp -a "spark-3.1.2-bin-hadoop3.2/bin/." "/usr/local/spark/bin/" &&     cp -a "spark-3.1.2-bin-hadoop3.2/jars/." "/usr/local/spark/assembly/target/scala-2.12/jars/" &&     rm "spark-3.1.2-bin-hadoop3.2.tgz":
#5 0.227 /bin/bash: line 1: wget: command not found
------
executor failed running [/bin/bash -o pipefail -o errexit -o nounset -o nolog -c cd "/tmp" &&     wget --no-verbose "https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" &&     tar -xvzf "spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" &&     mkdir -p "${SPARK_HOME}/bin" &&     mkdir -p "${SPARK_HOME}/assembly/target/scala-2.12/jars" &&     cp -a "spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}/bin/." "${SPARK_HOME}/bin/" &&     cp -a "spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}/jars/." "${SPARK_HOME}/assembly/target/scala-2.12/jars/" &&     rm "spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz"]: exit code: 127

请求解决该错误，并咨询：是否需要在容器中额外安装Python？因为Airflow使用Python开发。

错误解决 #

错误原因是apache/airflow:2.5.0基础镜像未预装wget工具，导致执行wget命令时找不到该命令。需要在安装Spark前先安装必要的系统工具。

修改后的Dockerfile如下：

FROM apache/airflow:2.5.0

ARG SPARK_VERSION="3.1.2"
ARG HADOOP_VERSION="3.2"

# 安装wget、tar等必要工具
RUN apt-get update && \
    apt-get install -y --no-install-recommends wget tar && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# setup spark
ENV SPARK_HOME /usr/local/spark
RUN cd "/tmp" && \
    wget --no-verbose "https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" && \
    tar -xvzf "spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" && \
    mkdir -p "${SPARK_HOME}/bin" && \
    mkdir -p "${SPARK_HOME}/assembly/target/scala-2.12/jars" && \
    cp -a "spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}/bin/." "${SPARK_HOME}/bin/" && \
    cp -a "spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}/jars/." "${SPARK_HOME}/assembly/target/scala-2.12/jars/" && \
    rm -rf "spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}" "spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz"

说明：

• 新增apt-get update && apt-get install -y --no-install-recommends wget tar命令，安装所需工具；
• 最后清理apt缓存并删除解压后的Spark目录，减少镜像体积。

Python安装疑问 #

不需要额外安装Python。apache/airflow官方镜像已经内置了对应版本的Python环境（Airflow 2.5.0默认使用Python 3.8），完全满足Airflow的运行需求，也能支持你编写的Python类型Airflow DAG。

内容的提问来源于stack exchange，提问作者KuRu