Kubernetes中Jenkins Pod持续Pending状态问题排查求助

问题现象 #

部署Jenkins应用后，Pod始终处于Pending状态，所有节点均显示Ready，但Pod无法完成调度。

已执行排查及输出 #

1. 节点状态检查：

$ kubectl get nodes
NAME          STATUS   ROLES           AGE   VERSION
server1       Ready    control-plane   8d    v1.24.9
server2       Ready    worker1         8d    v1.24.9
server3       Ready    worker2         8d    v1.24.9
server4       Ready    worker3         8d    v1.24.9

2. Jenkins namespace资源状态：

$ kubectl get all -n jenkins
NAME                          READY   STATUS    RESTARTS   AGE
pod/jenkins-6dc9f97c7-ttp64   0/1     Pending   0          7m42s

3. Pod调度事件详情：

$ kubectl describe pods jenkins-6dc9f97c7-ttp64 -n jenkins

Events:
Type     Reason            Age    From               Message
----     ------            ----   ----               -------
Warning  FailedScheduling  5m42s  default-scheduler  0/4 nodes are available: 3 node(s) had volume node affinity conflict, 4 node(s) didn't match Pod's node affinity/selector. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.

4. 节点标签详情：

$ kubectl get nodes --show-labels
NAME      STATUS   ROLES           AGE   VERSION   LABELS
server1   Ready    control-plane   9d    v1.24.9   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=server1,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=
server2   Ready    worker1         9d    v1.24.9   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=server2,kubernetes.io/os=linux,node-role.kubernetes.io/worker1=worker
server3   Ready    worker2         9d    v1.24.9   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=server3,kubernetes.io/os=linux,node-role.kubernetes.io/worker2=worker
server4   Ready    worker3         9d    v1.24.9   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=server4,kubernetes.io/os=linux,node-role.kubernetes.io/worker3=worker

5. 节点污点信息：

$ kubectl describe node | egrep -i taint
Taints:             key=value:NoSchedule
Taints:             <none>
Taints:             <none>
Taints:             <none>

当前配置情况 #

1. 更新后的Jenkins Deployment配置：

spec:
  nodeSelector:
    node-role.kubernetes.io/control-plane: ""
  tolerations:
  - key: node-role.kubernetes.io/control-plane

2. PersistentVolume配置片段：

...
local:
  path: /ksdata/apps/nodejs/
nodeAffinity:
  required:
    nodeSelectorTerms:
    - matchExpressions:
      - key: kubernetes.io/hostname
        operator: In
        values:
        - server1

原因分析 #

1. 节点亲和性与污点容忍不匹配：Deployment指定Pod调度到control-plane节点（即server1），但server1存在自定义污点key=value:NoSchedule，当前Deployment的tolerations仅包含对node-role.kubernetes.io/control-plane的容忍，未添加对该自定义污点的容忍规则，导致Pod无法调度到server1。
2. 存储卷节点亲和冲突：PersistentVolume绑定到server1节点，而worker节点（server2/server3/server4）不符合Deployment的nodeSelector规则，同时也无法挂载该PV，因此出现"volume node affinity conflict"。

解决方案 #

方案一：允许Pod调度到server1（control-plane节点） #

修改Deployment的tolerations，添加对server1自定义污点的容忍：

spec:
  nodeSelector:
    node-role.kubernetes.io/control-plane: ""
  tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule
  - key: key
    value: value
    operator: Equal
    effect: NoSchedule

更新配置后执行：

kubectl apply -f deployment.yml -n jenkins

方案二：将Pod调度到worker节点（推荐） #

如果不需要在control-plane节点运行Jenkins，可调整配置：

1. 修改Deployment，移除nodeSelector和control-plane相关的tolerations：

spec:
  # 移除nodeSelector配置
  # 移除control-plane相关tolerations

2. 调整PersistentVolume的nodeAffinity，将目标节点改为某个worker节点（例如server2）：

nodeAffinity:
  required:
    nodeSelectorTerms:
    - matchExpressions:
      - key: kubernetes.io/hostname
        operator: In
        values:
        - server2

或者使用支持动态分配的存储类替代本地PV，避免节点绑定限制。

更新配置后重新部署即可完成调度。

内容的提问来源于stack exchange，提问作者user4948798