Jenkins Kubernetes DevOps流水线执行失败排查求助

问题现象 #

DevOps流程频繁故障，Kubernetes中Pod几秒内就终止，无法查看完整日志。Jenkins控制台显示：获取Jenkinsfile后，创建的Pod中jnlp和my-agent容器均以退出码0终止，流水线未执行就中止，报错ID：org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: dfb66fca-365a-4a1f-ad39-a1aa21f15e1a。

控制台输出 #

Obtained Jenkinsfile from git https://github.com/yaohaihan/k8s-cicd-demo.git
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Created Pod: kubernates devops-test/github-k8s-cicd-demo-19-n1pjj-0x5r6-lcm44
devops-test/github-k8s-cicd-demo-19-n1pjj-0x5r6-lcm44 Container jnlp was terminated (Exit Code: 0, Reason: Completed)
devops-test/github-k8s-cicd-demo-19-n1pjj-0x5r6-lcm44 Container my-agent was terminated (Exit Code: 0, Reason: Completed)

- jnlp -- terminated (0)
-----Logs-------------
 -noreconnect                          : If the connection ends, don't retry
                                         and just exit. (default: false)
 -protocols VAL                        : Specify the remoting protocols to
                                         attempt when instanceIdentity is
                                         provided.
 -proxyCredentials USER:PASSWORD       : HTTP BASIC AUTH header to pass in for
                                         making HTTP authenticated proxy
                                         requests.
 -tunnel HOST:PORT                     : Connect to the specified host and
                                         port, instead of connecting directly
                                         to Jenkins. Useful when connection to
                                         Jenkins needs to be tunneled. Can be
                                         also HOST: or :PORT, in which case the
                                         missing portion will be auto-configured
                                         like the default behavior
 -url URL                              : Specify the Jenkins root URLs to
                                         connect to.
 -version                              : Shows the version of the remoting jar
                                         and then exits (default: false)
 -webSocket                            : Make a WebSocket connection to Jenkins
                                         rather than using the TCP port.
                                         (default: false)
 -webSocketHeader NAME=VALUE           : Additional WebSocket header to set, eg
                                         for authenticating with reverse
                                         proxies. To specify multiple headers,
                                         call this flag multiple times, one
                                         with each header
 -workDir FILE                         : Declares the working directory of the
                                         remoting instance (stores cache and
                                         logs by default)


- my-agent -- terminated (0)
-----Logs-------------
 -noreconnect                          : If the connection ends, don't retry
                                         and just exit. (default: false)
 -protocols VAL                        : Specify the remoting protocols to
                                         attempt when instanceIdentity is
                                         provided.
 -proxyCredentials USER:PASSWORD       : HTTP BASIC AUTH header to pass in for
                                         making HTTP authenticated proxy
                                         requests.
 -tunnel HOST:PORT                     : Connect to the specified host and
                                         port, instead of connecting directly
                                         to Jenkins. Useful when connection to
                                         Jenkins needs to be tunneled. Can be
                                         also HOST: or :PORT, in which case the
                                         missing portion will be auto-configured
                                         like the default behavior
 -url URL                              : Specify the Jenkins root URLs to
                                         connect to.
 -version                              : Shows the version of the remoting jar
                                         and then exits (default: false)
 -webSocket                            : Make a WebSocket connection to Jenkins
                                         rather than using the TCP port.
                                         (default: false)
 -webSocketHeader NAME=VALUE           : Additional WebSocket header to set, eg
                                         for authenticating with reverse
                                         proxies. To specify multiple headers,
                                         call this flag multiple times, one
                                         with each header
 -workDir FILE                         : Declares the working directory of the
                                         remoting instance (stores cache and
                                         logs by default)

[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Queue task was cancelled
org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: dfb66fca-365a-4a1f-ad39-a1aa21f15e1a
Finished: ABORTED

部分Pipeline代码 #

pipeline {
    agent {
        kubernetes {
                    yaml """
                    apiVersion: v1
                    kind: Pod
                    spec:
                      containers:
                      - name: my-agent
                        image: 192.168.110.122:8858/library/agent-maven:latest
                        imagePullPolicy: Always
                      - name: jnlp
                        image: jenkins/inbound-agent:4.10-3
                        args: ['\$(JENKINS_SECRET)', '\$(JENKINS_NAME)']
                        env:
                        - name: JENKINS_SECRET
                          valueFrom:
                            fieldRef:
                              fieldPath: metadata.annotations['jenkins.io/secret']
                        - name: JENKINS_NAME
                          valueFrom:
                            fieldRef:
                              fieldPath: metadata.annotations['jenkins.io/name']
                        - name: JENKINS_URL
                          value: "http://jenkins-service.devops-test.svc.cluster.local:8080/"
                    """
                    defaultContainer 'my-agent'

                }

    }

    parameters {
        gitParameter name: 'BRANCH_NAME', branch: '', branchFilter: '.*', defaultValue: 'master', description: '请选择要发布的分支', quickFilterEnabled: false, selectedValue: 'NONE', tagFilter: '*', type: 'PT_BRANCH'
        choice(name: 'NAMESPACE', choices: ['devops-dev', 'devops-test', 'devops-prod'], description: '命名空间')
        string(name: 'TAG_NAME', defaultValue: 'snapshot', description: '标签名称，必须以 v 开头，例如：v1、v1.0.0')
    }

    environment {
        DOCKER_CREDENTIAL_ID = 'harbor-user-pass'
        GIT_REPO_URL = 'https://github.com/yaohaihan/k8s-cicd-demo.git'
        GIT_CREDENTIAL_ID = 'github-user-pass'
        GIT_ACCOUNT = 'root' // change me
        KUBECONFIG_CREDENTIAL_ID = 'ec9a10b4-fa75-44bd-8832-0a5f1596479f'   cat ~/.kube/config，
        REGISTRY = '192.168.110.122:8858'
        DOCKERHUB_NAMESPACE = 'wolfcode' // change me
        APP_NAME = 'k8s-cicd-demo'
        SONAR_SERVER_URL = 'http://192.168.113.120:32276'
        SONAR_CREDENTIAL_ID = 'sonarqube-token'
    }

    stages {
        stage('unit test') {
            steps {
                sh 'mvn clean test'
            }
        }
    }
}

故障原因分析 #

1. jnlp容器参数解析失败：Pipeline中jnlp容器的args使用了转义的\$(JENKINS_SECRET)，在Groovy三重双引号的字符串中，这种转义会导致Kubernetes无法正确解析环境变量，容器启动时没有拿到有效的认证参数，默认输出帮助信息后退出（退出码0）。
2. my-agent容器无持久运行命令：自定义的agent-maven镜像可能基于Jenkins inbound-agent构建，默认启动命令是jnlp客户端，但未传入任何参数，因此同样输出帮助信息后退出，无法等待Jenkins下发构建任务。

解决方法 #

1. 修正jnlp容器的参数配置 #

去掉args中$的转义，让Kubernetes正确解析环境变量：

args: ['$(JENKINS_SECRET)', '$(JENKINS_NAME)']

2. 为my-agent容器添加持久运行命令 #

给my-agent容器指定command，让它保持运行状态，等待Jenkins执行任务：

- name: my-agent
  image: 192.168.110.122:8858/library/agent-maven:latest
  imagePullPolicy: Always
  command: ["sleep", "infinity"]

3. 验证镜像配置（可选） #

如果agent-maven镜像不需要默认的jnlp客户端行为，建议重新构建镜像，修改默认启动命令为sleep或其他持久运行的命令，避免容器自动退出。

修正后的完整agent配置示例 #

agent {
    kubernetes {
                yaml """
                apiVersion: v1
                kind: Pod
                spec:
                  containers:
                  - name: my-agent
                    image: 192.168.110.122:8858/library/agent-maven:latest
                    imagePullPolicy: Always
                    command: ["sleep", "infinity"]
                  - name: jnlp
                    image: jenkins/inbound-agent:4.10-3
                    args: ['$(JENKINS_SECRET)', '$(JENKINS_NAME)']
                    env:
                    - name: JENKINS_SECRET
                      valueFrom:
                        fieldRef:
                          fieldPath: metadata.annotations['jenkins.io/secret']
                    - name: JENKINS_NAME
                      valueFrom:
                        fieldRef:
                          fieldPath: metadata.annotations['jenkins.io/name']
                    - name: JENKINS_URL
                      value: "http://jenkins-service.devops-test.svc.cluster.local:8080/"
                """
                defaultContainer 'my-agent'
            }
}

验证方法 #

修改Pipeline后重新触发构建，查看Kubernetes Pod状态，确认jnlp和my-agent容器均处于Running状态，Jenkins流水线能正常执行unit test阶段的任务。

内容的提问来源于stack exchange，提问作者姚 Haihan