GitLab CI中coverage/pytest无输出及测试失败返回0退出码问题求助

1. 严重新问题：coverage run无输出且任务失败 #

更新依赖后，GitLab CI中执行coverage run ...无任何输出，后续coverage report提示"No data to report"，最终任务返回非零退出码失败。但coverage --help可正常运行，此前功能正常。

2. 轻微旧问题：测试失败时coverage仍返回0退出码 #

本地测试失败时coverage会返回非零退出码，但GitLab CI中即使测试失败，coverage仍返回0，该问题无法在本地复现。

精简后的.gitlab-ci.yml配置 #

workflow:
    rules:
        - if: $CI_COMMIT_REF_PROTECTED == "true"
        - if: $CI_PIPELINE_SOURCE == 'merge_request_event'

default:
    tags:
        - cvmfs
    before_script:
        - set +e
        - alias python3="python"
        - sudo mv CommonAnalysisFramework ..
        - export CAFPATH="$PWD/.."
        - ln -s templates/UserSettingsGuideline.py UserSettings.py
        - source $CAFPATH/CommonAnalysisFramework/${SETUP_SCRIPT}

variables:
    SETUP_SCRIPT: setup_root628.sh

image: atlas/centos7-atlasos-dev

stages:
  - build
  - run
  - check
  - lint

test:
    stage: check
    before_script:
        - set +e
        - alias python3="python"
        - sudo mv CommonAnalysisFramework ..
        - export CAFPATH="$PWD/.."
        - ln -s templates/UserSettingsGuideline.py UserSettings.py
        - source $CAFPATH/CommonAnalysisFramework/${SETUP_SCRIPT}
        - source /cvmfs/sft.cern.ch/lcg/releases/LCG_104/pytest/7.4.0/x86_64-centos7-gcc12-opt/pytest-env.sh
        - source /cvmfs/sft.cern.ch/lcg/releases/LCG_104/pytest_cov/3.0.0/x86_64-centos7-gcc12-opt/pytest_cov-env.sh
        - unalias python3
    script:
        - set +e
        - echo " in main script"
        - coverage run --branch --source="." --omit="*config*,test/*.py,*UserSettings*,templates/*,*__init__.py" -m pytest test
        - coverage report -m --ignore-errors
        - coverage html --ignore-errors
    artifacts:
        paths:
            - .coverage
            - htmlcov

CI执行输出 #

Getting source from Git repository
00:02
Fetching changes...
Initialized empty Git repository in /builds/atlas-germany-dresden-vbs-group/CAF-Scripts/.git/
Created fresh repository.
Checking out 3f716e4e as detached HEAD (ref is Root628)...
Skipping Git submodules setup
Downloading artifacts
00:01
Downloading artifacts for compile (32909051)...
Downloading artifacts from coordinator... ok        host=gitlab.cern.ch OK
Downloading artifacts for run_USGuideline (32909052)...
Downloading artifacts from coordinator... ok        host=gitlab.cern.ch OK
Executing "step_script" stage of the job script
00:16
$ # INFO: Lowering limit of file descriptors for backwards compatibility. ffi: https://cern.ch/gitlab-runners-limit-file-descriptors # collapsed multi-line command
$ set +e
$ alias python3="python"
$ sudo mv CommonAnalysisFramework ..
$ export CAFPATH="$PWD/.."
$ ln -s templates/UserSettingsGuideline.py UserSettings.py
$ source $CAFPATH/CommonAnalysisFramework/${SETUP_SCRIPT}
lsetup               lsetup <tool1> [ <tool2> ...] (see lsetup -h):
 lsetup asetup        (or asetup) to setup an Athena release
 lsetup astyle        ATLAS style macros
 lsetup atlantis      Atlantis: event display
 lsetup eiclient      Event Index 
 lsetup emi           EMI: grid middleware user interface 
 lsetup ganga         Ganga: job definition and management client
 lsetup lcgenv        lcgenv: setup tools from cvmfs SFT repository
 lsetup panda         Panda: Production ANd Distributed Analysis
 lsetup pyami         pyAMI: ATLAS Metadata Interface python client
 lsetup root          ROOT data processing framework
 lsetup rucio         distributed data management system client
 lsetup scikit        python data analysis ecosystem
 lsetup views         Set up a full LCG release
 lsetup xcache        XRootD local proxy cache
 lsetup xrootd        XRootD data access
advancedTools        advanced tools menu
diagnostics          diagnostic tools menu
helpMe               more help
printMenu            show this menu
showVersions         show versions of installed software
 
06 Mar 2023 
centos7: setupATLAS is python3 environment by default (same as setupATLAS -3).
         If you need the previous python2 environment, do setupATLAS -2.
     We strongly encourage API users to migrate their scripts to python3.
          * * * This environment has been setup as python3. * * *
$ source /cvmfs/sft.cern.ch/lcg/releases/LCG_104/pytest/7.4.0/x86_64-centos7-gcc12-opt/pytest-env.sh
$ source /cvmfs/sft.cern.ch/lcg/releases/LCG_104/pytest_cov/3.0.0/x86_64-centos7-gcc12-opt/pytest_cov-env.sh
$ unalias python3
$ set +e
$ echo " in main script"
 in main script
$ coverage run --branch --source="." --omit="*config*,test/*.py,*UserSettings*,templates/*,*__init__.py" -m pytest test
$ coverage report -m --ignore-errors
No data to report.
$ coverage html --ignore-errors
No data to report.
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: command terminated with exit code 1

示例测试文件 #

#!/bin/env python
"""Test that the nominal cut is part of the cut list.

Also requires it to not be commented out for tables."""

import templates.WZUserSettings as us

def test_cutNames():
    cutnames = us.cutNames
    defaultCut = us.defaultCut.replace("*", "")
    assert defaultCut in cutnames

排查思路与解决方案 #

针对问题1（coverage run无输出） #

1. 暴露错误细节

   • 去掉脚本中的set +e，让命令执行的错误信息直接输出，掩盖错误会导致无法定位coverage run的失败原因。
   • 在coverage run前添加版本检查命令：which python && python --version && coverage --version && pytest --version，确认工具路径和版本是否匹配，避免setup脚本覆盖了pytest/coverage的环境。

2. 拆分命令排查

   • 先单独执行pytest test，确认pytest本身能正常运行并输出测试结果；再单独执行coverage run -m pytest test（去掉多余的--branch、--source、--omit参数），逐步排查是否是参数配置错误导致无数据。

3. 检查路径与权限

   • 添加pwd && ls -la test命令，确认当前工作目录和测试文件路径是否正确，避免mv CommonAnalysisFramework等操作导致路径混乱。
   • 去掉sudo操作，检查是否是权限问题导致coverage无法生成.coverage数据文件。

4. 开启debug模式

   • 执行coverage run --debug=trace --branch ...，查看coverage的执行日志，定位数据收集阶段是否出现异常。

针对问题2（测试失败时coverage返回0） #

1. 控制退出码传递

   • 去掉set +e，让coverage run的退出码直接继承pytest的结果；或者在coverage run后保存退出码：EXIT_CODE=$?，后续根据该变量判断是否终止任务。
   • 检查CI脚本中是否有其他命令覆盖了退出码，比如coverage report --ignore-errors只会忽略自身错误，不影响前面命令的退出码，问题大概率出在set +e掩盖了pytest的失败信号。

2. 模拟CI环境复现

   • 本地使用atlas/centos7-atlasos-dev镜像创建容器，复制CI中的所有setup步骤，模拟CI环境尝试复现问题，确认是否是环境差异导致的异常。

内容的提问来源于stack exchange，提问作者J.N.