GitLab Runner test阶段完成后cache缺失war文件致部署失败问题咨询

我的gitlab-ci.yml配置了3个阶段，用于将应用部署到OKD Pod，该应用是运行在tomcat:8上的Spring Boot项目。偶现stage执行完成后cache.zip未更新的问题，导致后续步骤无法正常运行。

复现步骤 #

GitLab CI依次运行以下阶段：

• 阶段1：执行test compile -> 运行正常
• 阶段2：打包生成war文件作为部署输出 -> GitLab CI日志显示执行成功，但cache.zip中不存在war文件（仅偶发该问题，多数时候运行正常）
• 阶段3：将war文件部署到Pod -> 由于cache.zip中不存在war文件，脚本报错，任务失败

.gitlab-ci.yml配置 #

image: openshift/origin-cli

stages:
  - build
  - test
  - staging

cache:
  paths:
    - .m2/repository
    - target
    - artifact

validate:jdk8:
  stage: build
  script:
    - 'mvn test-compile'
  only:
    - master
  image: maven:3.3.9-jdk-8

verify:jdk8:
  stage: test
  script:
    - 'mvn verify'
    - 'mvn package' # 该命令生成war文件
  only:
    - master
  image: maven:3.3.9-jdk-8

staging:
  script:
    - "mkdir -p artifact"
    - "cp ./target/*.war ./artifact/" # 偶发该行报错，因为上一步没有把war文件写入缓存
    - "oc start-build $APP"
    - "rm -rf ./target/* && rm -rf ./artifact/*" # 移除war和class文件，仅缓存m2依赖
  stage: staging
  variables:
    APP: $CI_PROJECT_NAME
  environment:
    name: staging
    url: http://$CI_PROJECT_NAME-staging.$OPENSHIFT_DOMAIN
  only:
    - master

实际现象 #

偶发test阶段执行完成后cache中不存在war文件的问题（暂不确定是否和war文件大小有关）

预期现象 #

test阶段执行完成后war文件会更新到cache中，供后续staging阶段部署使用

相关日志与截图 #

错误截图已省略

任务日志 #

Running with gitlab-runner 13.7.0 (943fc252)
  on gitlab-runner-node1 y6awygsj
Preparing the "docker" executor
00:01
Using Docker executor with image openshift/origin-cli ...
Using locally found image version due to if-not-present pull policy
Using docker image sha256:7ebb6be01117a50344d63f77c385a13302afecd33480b97c36a518d4f5ebc25a for openshift/origin-cli with digest docker.io/openshift/origin-cli@sha256:509e052d0f2d531b666b7da9fa49c5558c76ce5d286456f0859c0a49b16d6bf2 ...
Preparing environment
00:00
Running on runner-y6awygsj-project-489-concurrent-0 via gitlab.runner.node1...
Getting source from Git repository
00:01
Fetching changes...
Reinitialized existing Git repository in /builds/my-project/.git/
Checking out b4c97428 as master...
Removing .m2/
Removing artifact/
Removing target/
Skipping Git submodules setup
Restoring cache
00:05
Checking cache for default-23...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Executing "step_script" stage of the job script
00:01
$ mkdir -p artifact
$ cp ./target/*.war ./artifact/
cp: cannot stat './target/*.war': No such file or directory
Cleaning up file based variables
00:00
ERROR: Job failed: exit code 1

环境说明 #

config.toml配置 #

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "gitlab-runner-node1"
  url = "https://gitlab.mycompany.vn/"
  token = "y6awygsj9zks18nU6PDt"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    dns = ["192.168.100.1"]
    tls_verify = false
    image = "alpine:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/mnt/nfs/nfsshare-gitlab/cache:/cache"]
    shm_size = 0
    pull_policy = "if-not-present"

GitLab Runner版本 #

• 版本: 13.7.0
• Git 修订版本: 943fc252
• Git 分支: 13-7-stable
• GO 版本: go1.13.8
• 构建时间: 2020-12-21T13:47:06+0000
• OS/架构: linux/amd64

临时解决方法 #

重新运行test阶段直到cache中包含war文件。

根因分析 #

该偶发问题核心是混用了GitLab CI的缓存和产物传递逻辑：

1. GitLab CI的cache设计初衷是缓存依赖（比如Maven依赖），并非用于跨阶段传递构建产物，缓存不保证100%在阶段间同步，多runner调度、缓存更新失败、命中旧版本缓存都会导致产物丢失
2. 当前全局缓存未配置自定义key，默认使用default作为缓存key，不同分支、不同提交的任务会互相覆盖缓存，可能出现test阶段刚生成的war包被旧缓存覆盖的情况
3. staging阶段最后清空了target和artifact目录，该操作也会被写入缓存，后续任务如果命中了清空后的缓存，自然找不到war包

修复方案 #

1. 推荐方案：用artifacts代替cache传递构建产物 #

artifacts是GitLab CI专门设计用于跨阶段传递产物的功能，可用性远高于cache，修改后核心配置如下：

image: openshift/origin-cli

stages:
  - build
  - test
  - staging

# cache仅保留maven依赖，不存放构建产物
cache:
  paths:
    - .m2/repository

validate:jdk8:
  stage: build
  script:
    - 'mvn test-compile'
  only:
    - master
  image: maven:3.3.9-jdk-8

verify:jdk8:
  stage: test
  script:
    - 'mvn verify'
    - 'mvn package'
  only:
    - master
  image: maven:3.3.9-jdk-8
  # 新增artifacts配置，存储war包为制品，保留1天足够部署使用
  artifacts:
    paths:
      - target/*.war
    expire_in: 1d

staging:
  script:
    - "mkdir -p artifact"
    - "cp ./target/*.war ./artifact/"
    - "oc start-build $APP"
  stage: staging
  variables:
    APP: $CI_PROJECT_NAME
  environment:
    name: staging
    url: http://$CI_PROJECT_NAME-staging.$OPENSHIFT_DOMAIN
  only:
    - master
  # 明确依赖test阶段，自动拉取对应制品
  dependencies:
    - verify:jdk8

2. 若坚持使用cache，可做如下调整 #

• 给缓存增加唯一key，使用提交哈希作为key，避免不同提交的缓存互相覆盖：

cache:
  key: "$CI_COMMIT_SHA"
  paths:
    - .m2/repository
    - target

• 删除staging阶段最后清空target和artifact目录的命令，避免清空操作更新缓存
• 给test阶段增加缓存策略policy: push-pull，强制生成的产物写入缓存

3. NFS缓存适配调整 #

从config.toml可知缓存存储在NFS共享存储上，可增加缓存压缩配置，避免大文件写入缓存失败：
在config.toml的[runners.cache]下新增配置：

[runners.cache]
  Type = "local"
  Path = "/cache"
  Shared = true
  CompressionLevel = "fastest"

内容的提问来源于stack exchange，提问作者Vu Hong Ha