GitLab CI/CD修改文件后构建Spark镜像内容异常问题

问题现象 #

修改Dockerfile本身或任何被复制到容器的文件后，GitLab CI/CD构建的镜像中/opt/spark/jars/目录仅包含自行添加的JAR包，缺失FROM spark:3.4.1-python3基础镜像自带的原Spark JAR包，导致Spark任务执行报错。

技术栈 #

• Kubernetes部署的Spark集群（1个Master节点 + 2个Worker节点）
• 本地Airflow向Spark集群提交任务
• 每个Airflow任务通过Docker Operator运行包含spark-submit的Docker容器
• GitLab CI/CD负责构建Docker镜像并推送到私有GitLab镜像仓库

.gitlab-ci.yml配置 #

build:
  stage: build
  when: manual
  only:
    - tags

  services:
    - docker:17.06.2-dind

  script:
    - docker build -q --no-cache --build-arg MINIO_KEY=$MINIO_KEY --build-arg MINIO_SECRET=$MINIO_SECRET -t $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG

Dockerfile配置 #

FROM spark:3.4.1-python3

USER root

RUN apt-get -qq update && apt-get -qq install -y python3-pip wget git

RUN wget -q https://mygit.com.br/api/v4/projects/341/repository/files/jars-spark.tar/raw?private_token=mytoken -O jars-spark.tar \
&& tar -xvf jars-spark.tar --strip-components=1 -C /opt/spark/jars jars-spark \
&& rm -f jars-spark.tar 

WORKDIR /opt/spark/conf
COPY ./log4j2.properties log4j2.properties

WORKDIR /opt/spark/app
COPY ./docrecon-landing-incremental.py docrecon-landing-incremental.py
COPY ./docrecon-bronze.py docrecon-bronze.py
COPY ./docrecon-silver.py docrecon-silver.py

COPY ./moviproc-landing-incremental.py moviproc-landing-incremental.py
COPY ./moviproc-bronze.py moviproc-bronze.py
COPY ./moviproc-silver.py moviproc-silver.py

COPY ./requirements.txt requirements.txt

RUN pip3 install --timeout 1000 -r requirements.txt --user

ARG MINIO_KEY
ENV MINIO_KEY=${MINIO_KEY}

ARG MINIO_SECRET
ENV MINIO_SECRET=${MINIO_SECRET}

错误表现 #

• 镜像内容差异：标签0.1.10的镜像包含基础镜像+自行添加的完整JAR包集合；修改文件后的标签0.1.11镜像仅存在自行添加的JAR包。
• Spark任务执行报错：

airflow.exceptions.AirflowException: Docker container failed: {'Error': None, 'StatusCode': 1} lines Error: Could not find or load main class org.apache.spark.launcher.Main
Caused by: java.lang.ClassNotFoundException: org.apache.spark.launcher.Main
/opt/spark/bin/spark-class: line 97: CMD: bad array subscript

排查信息 #

仅通过GitLab的dind构建部署镜像时会出现该问题，本地构建推送则无异常；GitLab Runner日志中存在的错误在镜像正常构建时也会出现，判断与当前问题无关。

内容的提问来源于stack exchange，提问作者jonasmuller98