同一K8s集群下Jenkins不同Pod配置中相同镜像行为不一致问题

我构建了一个Jenkins代理镜像并上传至Docker，Dockerfile内容如下：

FROM jenkins/inbound-agent:jdk11 as agent

LABEL maintainer="JoSSte <re@dacted>"

USER root
ENV TZ=Europe/Copenhagen
RUN ln -snf /usr/share/zoneinfo/"${TZ}" /etc/localtime && echo "${TZ}" > /etc/timezone \
&&  dpkg-reconfigure -f noninteractive tzdata 

# update apt cache
RUN apt-get update \
&&  apt-get upgrade -qy \
&&  apt-get install -y curl lsb-release ca-certificates apt-transport-https software-properties-common gnupg2

# add php8 repo
RUN curl -sSLo /usr/share/keyrings/deb.sury.org-php.gpg https://packages.sury.org/php/apt.gpg \
&&  sh -c 'echo "deb [signed-by=/usr/share/keyrings/deb.sury.org-php.gpg] https://packages.sury.org/php/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/php.list' \
&&  apt-get update

# install php8 & composer
RUN apt-get install -qy rsync zip php8.2-curl php8.2-gd apache2 php8.2 unzip php8.2-mysql php8.2-zip php8.2-mbstring php-xdebug php-pear* \
&&  curl -sSLo composer-setup.php https://getcomposer.org/installer \
&&  php composer-setup.php --install-dir=/usr/local/bin --filename=composer \
&&  composer self-update  

# Cleanup old packagess
RUN apt-get -qy autoremove

USER jenkins

ENTRYPOINT ["jenkins-slave"]

该镜像在Docker代理的Jenkins构建环境中运行正常。

迁移到Kubernetes后，先用podTemplate测试，脚本如下：

podTemplate(containers: [
    containerTemplate(name: 'php8builder', image: 'jstevnsvig/jenkins-build-slave-php:v8.2', command: 'sleep', args: '99d')
  ]) {

    node(POD_LABEL) {
        container('php8builder') {
            stage('list apt packages') {
                sh '/bin/sh -c "dpkg-query -l | cat"'
            }
            stage('dump php version') {
                sh '/bin/sh -c "php --version"'
            }
            stage('dump /usr/local/bin files') {
                sh 'ls /usr/local/bin'
            }
            stage('dump composer version') {
                sh '/bin/sh -c "composer --version"'
            }
        }
    }
}

测试结果正常，关键输出如下：

• /bin/sh -c php --version
  PHP 8.2.1 (cli) (built: Jan 13 2023 10:38:46) (NTS)
  Copyright (c) The PHP Group
  Zend Engine v4.2.1, Copyright (c) Zend Technologies
  with Zend OPcache v8.2.1, Copyright (c), by Zend Technologies
  with Xdebug v3.2.0, Copyright (c) 2002-2022, by Derick Rethans

• ls /usr/local/bin
  composer
  jenkins-agent
  jenkins-slave

• /bin/sh -c composer --version
  Composer version 2.5.1 2022-12-22 15:33:54

但改用Kubernetes YAML文件定义Pod运行流水线时，同一镜像在相同环境下找不到Composer。流水线脚本：

pipeline {
    agent { 
        kubernetes {
            yamlFile 'KubernetesPod.yaml'
            retries 2
          }
        }
    options {
        disableConcurrentBuilds()
    }
    stages {
        stage ('Staging'){
            steps {
                echo "looking for composer"
                sh 'ls /usr/local/bin'
            }
        }
        stage('run php') {
            steps {
                container('phpbuilder') {
                    sh 'php --version'
                    sh 'php src/test.php'
                }
            }
        }
    }
}

对应的KubernetesPod.yaml内容：

metadata:
  labels:
    some-label: some-label-value
spec:
  containers:
  - name: phpbuilder
    image: jstevnsvig/jenkins-build-slave-php:v8.2
    command:
    - sleep
    args:
    - 99d

运行后输出显示Composer缺失：

looking for composer

• ls /usr/local/bin
  jenkins-agent
  jenkins-slave

• php --version
  PHP 8.2.1 (cli) (built: Jan 13 2023 10:38:46) (NTS)
  Copyright (c) The PHP Group
  Zend Engine v4.2.1, Copyright (c) Zend Technologies
  with Zend OPcache v8.2.1, Copyright (c), by Zend Technologies
  with Xdebug v3.2.0, Copyright (c) 2002-2022, by Derick Rethans

已确认通过Jenkins云配置的Agent仍能正常运行，为何用Kubernetes YAML定义Pod会出现这个问题？

核心原因 #

问题出在Jenkins Kubernetes插件的默认行为：使用自定义YAML时，插件会自动为Pod添加一个jnlp容器作为主代理容器。你流水线的Staging阶段没有指定容器，默认会在jnlp容器中执行命令——这个容器用的是Jenkins默认代理镜像，里面没有安装Composer，所以看不到composer文件；而run php阶段明确指定了phpbuilder容器，所以PHP命令能正常执行。

解决方案 #

方案1：所有步骤明确指定容器 #

修改流水线脚本，让Staging阶段也在phpbuilder容器中执行：

pipeline {
    agent { 
        kubernetes {
            yamlFile 'KubernetesPod.yaml'
            retries 2
          }
        }
    options {
        disableConcurrentBuilds()
    }
    stages {
        stage ('Staging'){
            steps {
                container('phpbuilder') {
                    echo "looking for composer"
                    sh 'ls /usr/local/bin'
                }
            }
        }
        stage('run php') {
            steps {
                container('phpbuilder') {
                    sh 'php --version'
                    sh 'php src/test.php'
                }
            }
        }
    }
}

方案2：将自定义容器设为默认代理容器 #

把YAML中自定义容器的名称改为jnlp，让插件将其识别为主代理容器，这样未指定容器的步骤会默认使用它：

metadata:
  labels:
    some-label: some-label-value
spec:
  containers:
  - name: jnlp
    image: jstevnsvig/jenkins-build-slave-php:v8.2
    command:
    - sleep
    args:
    - 99d

方案3：禁用默认jnlp容器 #

在YAML中添加agent: false，告诉插件不要自动添加jnlp容器，同时将容器命令改回镜像原本的jenkins-slave，确保它能正常作为Jenkins代理运行：

metadata:
  labels:
    some-label: some-label-value
spec:
  agent: false
  containers:
  - name: phpbuilder
    image: jstevnsvig/jenkins-build-slave-php:v8.2
    command:
    - jenkins-slave
    args: []

内容的提问来源于stack exchange，提问作者JoSSte