GitLab CI拆分Docker构建为多阶段 实现依赖安装与前端构建并行

解决方案 #

可以实现，核心思路是拆分Docker镜像为依赖基础镜像和业务镜像两层，同时调整CI阶段让依赖安装和前端构建并行执行，具体改造步骤如下：

步骤1：拆分Dockerfile #

将原Dockerfile拆分为两个文件，分离依赖层和业务层：

1. 新建Dockerfile.base，存放所有依赖安装、基础环境配置逻辑（对应原Dockerfile中ADD app.tar /app/之前的全部内容）：

FROM python:3.7-slim

# Add user
ARG APP_USER=abc
RUN groupadd -r ${APP_USER} && useradd --no-log-init -r -g ${APP_USER} ${APP_USER}

WORKDIR /app
ENV PYTHONUNBUFFERED=1
EXPOSE 80
EXPOSE 8080

ADD requirements.txt /app/

RUN set -ex \
    && BUILD_DEPS=" \
        gcc \
    " \
    && RUN_DEPS=" \
        ffmpeg \
        postgresql-client \
        nginx \
        dumb-init \
    " \
    && apt-get update && apt-get install -y $BUILD_DEPS \
    && pip install --no-cache-dir --default-timeout=100000 -r /app/requirements.txt \
    && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false  $BUILD_DEPS \
    && apt-get update && apt-get install -y --no-install-recommends $RUN_DEPS \
    && rm -rf /var/lib/apt/lists/*

# Set uWSGI settings
ENV UWSGI_WSGI_FILE=/app/api/api/wsgi.py UWSGI_HTTP=:8000 UWSGI_MASTER=1 UWSGI_HTTP_AUTO_CHUNKED=1 UWSGI_HTTP_KEEPALIVE=1 UWSGI_LAZY_APPS=1 UWSGI_WSGI_ENV_BEHAVIOR=holy PYTHONUNBUFFERED=1 UWSGI_WORKERS=2 UWSGI_THREADS=4
ENV UWSGI_STATIC_EXPIRES_URI="/static/.*\.[a-f0-9]{12,}\.(css|js|png|jpg|jpeg|gif|ico|woff|ttf|otf|svg|scss|map|txt) 315360000"
ENV PYTHONPATH=$PYTHONPATH:/app/api:/app
ENV DB_PORT=5432 DB_NAME=shittywizard DB_USER=shittywizard DB_HOST=localhost

ADD nginx.conf /etc/nginx/nginx.conf

# Set entrypoint
ADD entrypoint.sh /
RUN chmod 755 /entrypoint.sh
ENTRYPOINT ["dumb-init", "--", "/entrypoint.sh"]

2. 新建Dockerfile.final，仅包含业务代码导入和后续收尾步骤：

FROM $CI_REGISTRY_IMAGE/base:latest

ADD app.tar /app/
RUN python /app/api/manage.py collectstatic --noinput

步骤2：调整gitlab-ci.yml配置 #

修改阶段定义，将原来的串行两个阶段调整为「预构建并行阶段」+「最终镜像构建阶段」，完整配置如下：

stages:
  - Pre Build
  - Build Final Image

services:
  - docker:18.03-dind

variables:
  DOCKER_DRIVER: overlay2
  DOCKER_HOST: tcp://localhost:2375
  DOCKER_TLS_CERTDIR: ""

# 前端构建任务，和基础镜像构建并行执行
build_node:
  stage: Pre Build
  only:
    - staging
    - production
  image: node:14.8.0
  variables:
    GIT_SUBMODULE_STRATEGY: recursive
  artifacts:
    paths:
      - http
  cache:
    key: "node_modules"
    paths:
      - frontend/node_modules
  script:
    - cd frontend
    - yarn install --network-timeout 100000
    - CI=false yarn build
    - mv build ../http

# 基础镜像构建任务，仅处理apt、pip依赖安装，和前端构建并行
build_base_image:
  stage: Pre Build
  only:
    - staging
    - production
  image: docker
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    # 开启缓存复用，无变更时不需要重打基础镜像
    - docker pull $CI_REGISTRY_IMAGE/base:latest || true
    - docker build --network=host --cache-from $CI_REGISTRY_IMAGE/base:latest -f Dockerfile.base --tag $CI_REGISTRY_IMAGE/base:latest .
    - docker push $CI_REGISTRY_IMAGE/base:latest
  # 可选优化：仅依赖相关文件变更时才执行构建，否则直接跳过
  rules:
    - changes:
        - requirements.txt
        - Dockerfile.base
        - nginx.conf
        - entrypoint.sh
      when: always
    - when: always

# 最终镜像构建，依赖前面两个并行任务的输出
build_image:
  stage: Build Final Image
  only:
    - staging
    - production
  image: docker
  dependencies:
    - build_node
  script:
    - tar -cvf app.tar api/ discordbot/ helpers/ http/
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker pull $CI_REGISTRY_IMAGE:latest || true
    - docker build --network=host --cache-from $CI_REGISTRY_IMAGE:latest -f Dockerfile.final --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA --tag $CI_REGISTRY_IMAGE:latest .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    - docker push $CI_REGISTRY_IMAGE:latest

效果说明 #

• apt、pip依赖安装步骤完全和前端构建并行执行，整体CI耗时直接减少了原来串行的依赖安装时长
• 依赖无变更时，基础镜像直接复用历史镜像，整个CI流程仅需执行前端构建+最终镜像打包两步，耗时可缩短至原来的50%以上

内容的提问来源于stack exchange，提问作者cclloyd