GitLab CI/CD非main分支dev-deploy重复推送无流水线问题排查

流水线阶段与核心需求 #

我的GitLab CI/CD流水线包含4个阶段：

• dev-deploy
• test-deploy
• approval
• prod-deploy

dev-deploy阶段下有4个作业，需满足以下规则：

• 除默认分支main外，所有分支推送时仅生成包含dev-deploy阶段的手动流水线，不触发其他阶段
• dev-deploy阶段的作业禁止在任何MR事件中运行

当前配置的问题 #

目前为dev-deploy下所有作业配置的规则如下：

rules:
  - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
    when: manual
    allow_failure: false
  - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    when: never

该规则仅在分支首次从本地推送到远程时生效（比如创建本地分支x/foo并首次推送后，能生成含dev-deploy阶段的流水线），但同一分支后续推送新提交时，无法生成对应流水线。

完整作业配置 #

以下是dev-deploy阶段4个作业的完整配置：

dev-deploy-common:
  stage: dev-deploy
  script:
    - cd ./aws_code_base/platform/CommonLib/
    - python setup.py setopt --command=options.package_data --option=CommonLib --set-value=devConfigs.json
    - python setup.py sdist
  artifacts:
    name: dev-commonlib
    paths:
      - aws_code_base/platform/CommonLib/dist/*tar.gz
    expire_in: 1 day
  rules:
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: false
      changes:
        - aws_code_base/platform/emr_on_eks/Dockerfile
        - aws_code_base/platform/emr_on_eks/Notebook.Dockerfile
        - aws_code_base/platform/CommonLib/**
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: never
  tags:
    - dev
    - aws

dev-deploy-emrcustomimage:
  stage: dev-deploy
  script:
    - cd ./aws_code_base/platform/
    - aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin 123********.dkr.ecr.us-west-2.amazonaws.com
    - docker pull 123********.dkr.ecr.us-west-2.amazonaws.com/spark/emr-6.11.0:latest
    - docker pull 123********.dkr.ecr.us-west-2.amazonaws.com/notebook-jeg/emr-6.11.0:latest
    - docker pull 123********.dkr.ecr.us-west-2.amazonaws.com/notebook-spark/emr-6.11.0:latest
    - cp $CI_PROJECT_DIR/aws_code_base/platform/CommonLib/dist/*tar.gz .
    - docker build --no-cache -f ./emr_on_eks/Dockerfile -t emr6.11.0_custom .
    - docker build --no-cache -f ./emr_on_eks/Notebook.Dockerfile -t emr6.11.0_notebook_custom .
    - aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin {account_id}.dkr.ecr.us-west-2.amazonaws.com
    - docker tag emr6.11.0_custom {account_id}.dkr.ecr.us-west-2.amazonaws.com/emr6.11.0_custom_repo:dev
    - docker tag emr6.11.0_notebook_custom {account_id}.dkr.ecr.us-west-2.amazonaws.com/emr6.11.0_custom_repo:notebook_dev
    - docker push {account_id}.dkr.ecr.us-west-2.amazonaws.com/emr6.11.0_custom_repo:dev
    - docker push {account_id}.dkr.ecr.us-west-2.amazonaws.com/emr6.11.0_custom_repo:notebook_dev
    - IMAGES_TO_DELETE=$( aws ecr list-images --region us-west-2 --repository-name emr6.11.0_custom_repo --filter "tagStatus=UNTAGGED" --query 'imageIds[*]' --output json )
    - aws ecr batch-delete-image --region us-west-2 --repository-name emr6.11.0_custom_repo --image-ids "$IMAGES_TO_DELETE" || true
    - docker system prune -a -f
  rules:
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: false
      changes:
        - aws_code_base/platform/emr_on_eks/Dockerfile
        - aws_code_base/platform/emr_on_eks/Notebook.Dockerfile
        - aws_code_base/platform/CommonLib/**
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: never
  needs:
    - job: deploy-dev-commonlib
      artifacts: true
  tags:
    - dev
    - aws

dev-deploy-postgres:
  stage: dev-deploy
  script:
    - pip3 install boto3
    - git fetch
    - added_files=$(git diff-tree --no-commit-id --name-only -r $CI_COMMIT_SHA | tr '\n' ' ')
    - modified_files=$(git diff-tree --no-commit-id --name-only -r $CI_COMMIT_SHA | tr '\n' ' ')
    - renamed_files=$(git diff-tree --no-commit-id --name-only -r $CI_COMMIT_SHA | tr '\n' ' ')
    - echo $added_files "," $modified_files "," $renamed_files
    - python3 aws_code_base/platform/deployments/scripts/postgres_deploy.py "$added_files" "$modified_files" "$renamed_files" "aws_code_base/platform/CommonLib/CommonLib/devConfigs.json"
  rules:
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: false
      changes:
        - 'aws_code_base/platform/postgres/**'
        - 'aws_code_base/dataengineering/**/schema_*.json'
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: never
  tags:
    - dev
    - aws

dev-deploy-snowflake:
  stage: dev-deploy
  script:
    - pip3 install boto3 snowflake-connector-python --use-feature=2020-resolver
    - git fetch
    - added_files=$(git diff-tree --no-commit-id --name-only -r $CI_COMMIT_SHA | tr '\n' ' ')
    - modified_files=$(git diff-tree --no-commit-id --name-only -r $CI_COMMIT_SHA | tr '\n' ' ')
    - renamed_files=$(git diff-tree --no-commit-id --name-only -r $CI_COMMIT_SHA | tr '\n' ' ')
    - echo $added_files "," $modified_files "," $renamed_files
    - python3 aws_code_base/platform/deployments/scripts/snowflake_deploy.py "$added_files" "$modified_files" "$renamed_files" "aws_code_base/platform/CommonLib/CommonLib/devConfigs.json" "dev"
  rules:
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: false
      changes:
        - aws_code_base/snowflake/**
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: never
  tags:
    - dev
    - aws

内容的提问来源于stack exchange，提问作者techgeek11