僵尸Kubernetes Pod被ReplicaSet反复重启问题排查求助

云上老周

2026-8-10

Kubernetes Deployment副本数异常：期望1个却生成2个且修改不生效的排查方案

问题概述

cash-idemix-ca Deployment配置的期望副本数为1，但实际运行2个Pod
删除关联ReplicaSet后会被Deployment自动重建，无法将ReplicaSet副本数设为0
使用Lens修改Deployment副本数为1后无法持久生效，界面仍显示期望副本数为2

排查与解决步骤

1. 检查是否存在控制器冲突

确认是否有其他组件在自动修改Deployment副本数：

查看是否有Horizontal Pod Autoscaler（HPA）：
```
kubectl get hpa -n boc-cash-portal
```

检查Deployment注解与Helm同步状态：

kubectl describe deployment cash-idemix-ca -n boc-cash-portal | grep -A 10 "Annotations"
helm history cash-idemix-ca -n boc-cash-portal

排查CI/CD流水线是否有自动重新部署的任务。

2. 处理Deployment滚动更新超时

从提供的Deployment状态可见ProgressDeadlineExceeded，说明滚动更新超时导致旧ReplicaSet未被清理：

查看异常Pod的状态与事件：

kubectl get pods -n boc-cash-portal -l app.kubernetes.io/name=cash-idemix-ca
kubectl describe pod <异常Pod名称> -n boc-cash-portal

检查Pod无法正常终止的原因：比如PVC挂载无法卸载、容器进程不响应终止信号等。

3. 强制修复副本数并触发更新

确认无其他控制器干预后，手动强制同步Deployment状态：

重置副本数并触发滚动更新：

kubectl scale deployment cash-idemix-ca --replicas=1 -n boc-cash-portal
kubectl patch deployment cash-idemix-ca -n boc-cash-portal -p '{"spec":{"template":{"metadata":{"annotations":{"force-update":"'$(date +%s)'"}}}}}'

清理未被管理的旧ReplicaSet：

kubectl get rs -n boc-cash-portal -l app.kubernetes.io/name=cash-idemix-ca
# 删除当前活跃RS之外的其他旧RS
kubectl delete rs <旧RS名称> -n boc-cash-portal

4. 验证Lens界面与权限问题

用命令行确认实际副本数，排除Lens界面缓存问题：
```
kubectl get deployment cash-idemix-ca -n boc-cash-portal
```
检查Lens使用的账号是否有足够的Deployment修改权限，确认RBAC规则无限制。

5. 检查Kubernetes控制器状态

查看kube-controller-manager日志，确认Deployment控制器是否存在异常：

# kubeadm部署的集群示例
kubectl logs -n kube-system kube-controller-manager-<节点名称> | grep -i deployment

附：相关资源配置

ReplicaSet 配置

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: cash-idemix-ca-6bc646cbcc
  namespace: boc-cash-portal
  uid: babc0236-2053-4088-b8e8-b8ae2ed9303c
  resourceVersion: '25466073'
  generation: 1
  creationTimestamp: '2022-11-28T13:18:42Z'
  labels:
    app.kubernetes.io/instance: cash-idemix-ca
    app.kubernetes.io/name: cash-idemix-ca
    pod-template-hash: 6bc646cbcc
  annotations:
    deployment.kubernetes.io/desired-replicas: '1'
    deployment.kubernetes.io/max-replicas: '2'
    deployment.kubernetes.io/revision: '7'
  ownerReferences:
    - apiVersion: apps/v1
      kind: Deployment
      name: cash-idemix-ca
      uid: 2a3300ed-f666-4a30-98b7-7ab2ebcb2a0d
      controller: true
      blockOwnerDeletion: true
  managedFields:
    - manager: kube-controller-manager
      operation: Update
      apiVersion: apps/v1
      time: '2022-11-28T13:18:42Z'
      fieldsType: FieldsV1
    - manager: kube-controller-manager
      operation: Update
      apiVersion: apps/v1
      time: '2022-11-29T13:27:37Z'
      fieldsType: FieldsV1
      subresource: status
  selfLink: >-
    /apis/apps/v1/namespaces/boc-cash-portal/replicasets/cash-idemix-ca-6bc646cbcc
status:
  replicas: 1
  fullyLabeledReplicas: 1
  observedGeneration: 1
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: cash-idemix-ca
      app.kubernetes.io/name: cash-idemix-ca
      pod-template-hash: 6bc646cbcc
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: cash-idemix-ca
        app.kubernetes.io/name: cash-idemix-ca
        pod-template-hash: 6bc646cbcc
      annotations:
        kubectl.kubernetes.io/restartedAt: '2022-07-19T11:45:27Z'
    spec:
      volumes:
        - name: fabric-ca-server-home
          persistentVolumeClaim:
            claimName: cash-idemix-ca-fabric-ca-server-home
      containers:
        - name: cash-idemix-ca
          image: ca.icr.io/samara-dev-container-images/cash-idemix-ca:0.3.0
          command:
            - sh
          args:
            - '-c'
            - >-
              sleep 1 && fabric-ca-server start -b
              $(IDEMIX_ADMIN_USERNAME):$(IDEMIX_ADMIN_PASSWORD) --port 7054
              --idemix.curve gurvy.Bn254 --loglevel debug
          ports:
            - name: api
              containerPort: 7054
              protocol: TCP
          env:
            - name: FABRIC_CA_SERVER_HOME
              value: /idemix-config/fabric-ca-gurvy
            - name: IDEMIX_ADMIN_USERNAME
              valueFrom:
                secretKeyRef:
                  name: cash-idemix-ca-admin-credentials
                  key: IDEMIX_ADMIN_USERNAME
            - name: IDEMIX_ADMIN_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: cash-idemix-ca-admin-credentials
                  key: IDEMIX_ADMIN_PASSWORD
          resources: {}
          volumeMounts:
            - name: fabric-ca-server-home
              mountPath: /idemix-config/fabric-ca-gurvy
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: cash-idemix-ca
      serviceAccount: cash-idemix-ca
      securityContext: {}
      imagePullSecrets:
        - name: all-icr-io
      schedulerName: default-scheduler

Deployment 配置

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cash-idemix-ca
  namespace: boc-cash-portal
  uid: 2a3300ed-f666-4a30-98b7-7ab2ebcb2a0d
  resourceVersion: '25467341'
  generation: 10
  creationTimestamp: '2022-07-18T14:13:57Z'
  labels:
    app.kubernetes.io/instance: cash-idemix-ca
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: cash-idemix-ca
    app.kubernetes.io/version: manual-0.2.1
    helm.sh/chart: cash-idemix-ca-0.12.0
  annotations:
    deployment.kubernetes.io/revision: '7'
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"cash-idemix-ca","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"cash-idemix-ca","app.kubernetes.io/version":"manual-0.2.1","helm.sh/chart":"cash-idemix-ca-0.12.0"},"name":"cash-idemix-ca","namespace":"boc-cash-portal"},"spec":{"replicas":1,"revisionHistoryLimit":0,"selector":{"matchLabels":{"app.kubernetes.io/instance":"cash-idemix-ca","app.kubernetes.io/name":"cash-idemix-ca"}},"template":{"metadata":{"labels":{"app.kubernetes.io/instance":"cash-idemix-ca","app.kubernetes.io/name":"cash-idemix-ca"}},"spec":{"containers":[{"args":["-c","sleep
      1 \u0026\u0026 fabric-ca-server start -b
      $(IDEMIX_ADMIN_USERNAME):$(IDEMIX_ADMIN_PASSWORD) --port 7054
      --idemix.curve gurvy.Bn254 --loglevel
      debug"],"command":["sh"],"env":[{"name":"FABRIC_CA_SERVER_HOME","value":"/idemix-config/fabric-ca-gurvy"},{"name":"IDEMIX_ADMIN_USERNAME","valueFrom":{"secretKeyRef":{"key":"IDEMIX_ADMIN_USERNAME","name":"cash-idemix-ca-admin-credentials"}}},{"name":"IDEMIX_ADMIN_PASSWORD","valueFrom":{"secretKeyRef":{"key":"IDEMIX_ADMIN_PASSWORD","name":"cash-idemix-ca-admin-credentials"}}}],"image":"ca.icr.io/samara-dev-container-images/cash-idemix-ca:0.3.0","imagePullPolicy":"IfNotPresent","name":"cash-idemix-ca","ports":[{"containerPort":7054,"name":"api","protocol":"TCP"}],"volumeMounts":[{"mountPath":"/idemix-config/fabric-ca-gurvy","name":"fabric-ca-server-home","readOnly":false}]}],"imagePullSecrets":[{"name":"all-icr-io"}],"serviceAccountName":"cash-idemix-ca","volumes":[{"name":"fabric-ca-server-home","persistentVolumeClaim":{"claimName":"cash-idemix-ca-fabric-ca-server-home"}}]}}}}
status:
  observedGeneration: 10
  replicas: 2
  updatedReplicas: 1
  readyReplicas: 1
  availableReplicas: 1
  unavailableReplicas: 1
  conditions:
    - type: Available
      status: 'True'
      lastUpdateTime: '2022-11-28T13:53:56Z'
      lastTransitionTime: '2022-11-28T13:53:56Z'
      reason: MinimumReplicasAvailable
      message: Deployment has minimum availability.
    - type: Progressing
      status: 'False'
      lastUpdateTime: '2022-11-29T13:37:38Z'
      lastTransitionTime: '2022-11-29T13:37:38Z'
      reason: ProgressDeadlineExceeded
      message: ReplicaSet "cash-idemix-ca-6bc646cbcc" has timed out progressing.
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: cash-idemix-ca
      app.kubernetes.io/name: cash-idemix-ca
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: cash-idemix-ca
        app.kubernetes.io/name: cash-idemix-ca
      annotations:
        kubectl.kubernetes.io/restartedAt: '2022-07-19T11:45:27Z'
    spec:
      volumes:
        - name: fabric-ca-server-home
          persistentVolumeClaim:
            claimName: cash-idemix-ca-fabric-ca-server-home
      containers:
        - name: cash-idemix-ca
          image: ca.icr.io/samara-dev-container-images/cash-idemix-ca:0.3.0
          command:
            - sh
          args:
            - '-c'
            - >-
              sleep 1 && fabric-ca-server start -b
              $(IDEMIX_ADMIN_USERNAME):$(IDEMIX_ADMIN_PASSWORD) --port 7054
              --idemix.curve gurvy.Bn254 --loglevel debug
          ports:
            - name: api
              containerPort: 7054
              protocol: TCP
          env:
            - name: FABRIC_CA_SERVER_HOME
              value: /idemix-config/fabric-ca-gurvy
            - name: IDEMIX_ADMIN_USERNAME
              valueFrom:
                secretKeyRef:
                  name: cash-idemix-ca-admin-credentials
                  key: IDEMIX_ADMIN_USERNAME
            - name: IDEMIX_ADMIN_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: cash-idemix-ca-admin-credentials
                  key: IDEMIX_ADMIN_PASSWORD
          resources: {}
          volumeMounts:
            - name: fabric-ca-server-home
              mountPath: /idemix-config/fabric-ca-gurvy
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: cash-idemix-ca
      serviceAccount: cash-idemix-ca
      securityContext: {}
      imagePullSecrets:
        - name: all-icr-io
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 0
  progressDeadlineSeconds: 600

内容的提问来源于stack exchange，提问作者Herman

僵尸Kubernetes Pod被ReplicaSet反复重启问题排查求助

问题概述 #

排查与解决步骤 #

1. 检查是否存在控制器冲突 #

2. 处理Deployment滚动更新超时 #

3. 强制修复副本数并触发更新 #

4. 验证Lens界面与权限问题 #

5. 检查Kubernetes控制器状态 #

附：相关资源配置 #

ReplicaSet 配置 #

Deployment 配置 #