Azure Pipelines+SSH部署至集群首个可用VM的问题及方案咨询

环境与现有部署流程 #

我们使用本地Docker Swarm VM集群，搭配Azure Pipelines作为CI/CD工具。除main分支外，其他环境均采用单VM部署，流程运行顺畅；main分支需要部署至多台VM，当前流程为通过SSH将部署文件复制到VM，再执行部署命令。

当前Azure Pipelines配置 #

- job: Deploy_to_VM1
  condition: eq(variables['Build.SourceBranch'], 'refs/heads/main')
  dependsOn: 
    - Download_Artifact_and_Version
  variables:
    version: $[ dependencies.Download_Artifact_and_Version.outputs['setVersionVar.var'] ]
  displayName: 'Deploy to VM1'
  steps:
    - task: CopyFilesOverSSH@0
      displayName: 'Upload service files to vm1'
      inputs:
        sshEndpoint: 'sshvm1'
        sourceFolder: '$(System.DefaultWorkingDirectory)/docker-compose'
        contents: |
          config.yml
          docker-compose.yml
        targetFolder: '/tmp'
        cleanTargetFolder: false
        overwrite: true
        readyTimeout: '20000'

    - task: SSH@0
      inputs:
        sshEndpoint: 'sshvm1'
        runOptions: 'inline'
        inline: |
          export version=$(version)
          echo "$(REGISTRY_PASSWORD)" | docker login -u $(REGISTRY_USER) --password-stdin $(REGISTRY_URL)
          docker stack deploy -c /tmp/docker-compose.yml service
      condition: succeeded()
      name: deploy_vm1

- job: Deploy_to_VM2
  dependsOn: 
    - Download_Artifact_and_Version
    - Deploy_to_VM1
  condition: eq(dependencies.Deploy_to_VM1.result, 'Failed')
  variables:
    version: $[ dependencies.Download_Artifact_and_Version.outputs['setVersionVar.var'] ]
  displayName: 'Deploy to VM2'
  steps:
    - task: CopyFilesOverSSH@0
      displayName: 'Upload service files to vm2'
      inputs:
        sshEndpoint: 'sshvm2'
        sourceFolder: '$(System.DefaultWorkingDirectory)/docker-compose'
        contents: |
          config.yml
          docker-compose.yml
        targetFolder: '/tmp'
        cleanTargetFolder: false
        overwrite: true
        readyTimeout: '20000'

    - task: SSH@0
      inputs:
        sshEndpoint: 'sshvm2'
        runOptions: 'inline'
        inline: |
          export version=$(version)
          echo "$(REGISTRY_PASSWORD)" | docker login -u $(REGISTRY_USER) --password-stdin $(REGISTRY_URL)  
          docker stack deploy -c /tmp/docker-compose.yml service
      condition: succeeded()
      name: deploy_vm2

历史警告问题 #

此前Pipeline运行成功，但会输出两条被标记为错误级别的警告：

##[error]WARNING! Your password will be stored unencrypted in /***/.docker/config.json.
##[error]Since --detach=false was not specified, tasks will be created in the background. In a future release, --detach=false will become the default.

新部署异常现象 #

更换VM后出现两类异常：

• 部分服务仅首个可用VM部署成功，Pipeline状态显示「部分成功」
• 部分场景下，VM1实际部署成功，但被Pipeline标记为失败，进而触发VM2重复部署，所有VM均出现该误判重复部署情况

咨询要点 #

1. 上述部署异常的原因分析
2. 针对本地VM集群通过SSH部署的更优策略
3. Strategy Job的适用性：如何实现首个VM部署成功后停止后续部署，以及动态传入SSH服务连接的可行性

内容的提问来源于stack exchange，提问作者Spyros Tserentzoulias