Kubernetes CronJob Pod长期处于Terminating状态问题求助

API 探路者

2026-6-21

Kubernetes CronJob Pod 无法正常终止问题排查

问题现象

我有一个Kubernetes CronJob，每15分钟执行一次脚本将Git仓库克隆到持久存储中。但每隔1-2天会出现Job无法正常结束的情况：Pod处于Terminating状态，且日志确认脚本已完成所有任务。常规删除Pod命令会挂起，只有强制删除才能生效。

相关事件日志

│   Warning  FailedKillPod  14h (x94 over 2d10h)       kubelet  error killing pod: [failed to "KillContainer" for "pull-repo" with KillContainerError: "rpc error: code = DeadlineExceeded desc = an error occurs during waiting for container \"3ff92cf6a48e46345e635a5bbc8f20d334c84927b7e700076f9c795799029d19\" to be killed: wait container \"3ff92cf6a48e46345e635a5bbc8f20d334c84927b7e700076f9c795799029d19\": context deadline exceeded", failed to "KillPodSandbox" for "34ceec99-8a5e-45b8-a298-cdf2ae8e43df" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"]                                                                                                                                                                                                                                      │
│   Warning  FailedKillPod  7h20m (x110 over 2d10h)    kubelet  error killing pod: [failed to "KillContainer" for "pull-repo" with KillContainerError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded", failed to "KillPodSandbox" for "34ceec99-8a5e-45b8-a298-cdf2ae8e43df" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = failed to stop container \"3ff92cf6a48e46345e635a5bbc8f20d334c84927b7e700076f9c795799029d19\": an error occurs during waiting for container \"3ff92cf6a48e46345e635a5bbc8f20d334c84927b7e700076f9c795799029d19\" to be killed: wait container \"3ff92cf6a48e46345e635a5bbc8f20d334c84927b7e700076f9c795799029d19\": context deadline exceeded"]                                                                                                                                       │
│   Warning  FailedKillPod  5h49m (x446 over 2d10h)    kubelet  error killing pod: [failed to "KillContainer" for "pull-repo" with KillContainerError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded", failed to "KillPodSandbox" for "34ceec99-8a5e-45b8-a298-cdf2ae8e43df" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"]                                                                                                                                                                       │
│   Warning  Unhealthy      4m52s (x21033 over 2d10h)  kubelet  (combined from similar events): Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "1498eacd7dc6dfb402d060fb64114336fb9994060c9ac718aea16c3bd92f6fd8": OCI runtime exec failed: exec failed: cannot exec in a stopped container: unknown

CronJob 定义

apiVersion: batch/v1
kind: CronJob
metadata:
  name: job-name
spec:
  schedule: "*/15 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      parallelism: 1
      completions: 1
      backoffLimit: 3
      activeDeadlineSeconds: 120
      ttlSecondsAfterFinished: 10800
      template:
        spec:
          serviceAccountName: job-sa
          containers:
            - name: name
              image: image_name:1.00.00
              resources:
                requests:
                  memory: "150Mi"
                  cpu: "25m"
                limits:
                  memory: "300Mi"
                  cpu: "50m"
              livenessProbe:
                exec:
                  command: ["ls"]
              readinessProbe:
                exec:
                  command: ["ls"]
              command: ["/bin/sh"]
              args:
                - -c
                - /scripts/pull-repo.sh |& while IFS= read -r line; do printf '[%s] %s\n' "$(date '+%Y-%m-%dT%H:%M:%S.%N%:z')" "$line"; done |& tee -a /opt/logs/app/pull-repo/pull-repo.log
              volumeMounts:
                - name: config-server-repo-volume
                  mountPath: /tmp/git_repo_clone
                - name: git-repo-ssh-volume
                  mountPath: /opt/app-root/src/.ssh/config
                  subPath: config
                - name: git-repo-ssh-volume
                  mountPath: /opt/app-root/src/.ssh/id_rsa
                  subPath: id_rsa
                - name: git-repo-ssh-volume
                  mountPath: /opt/app-root/src/.ssh/known_hosts
                  subPath: known_hosts
                - name: scripts-volume
                  mountPath: /scripts/pull-repo.sh
                  subPath: pull-repo.sh
          volumes:
            - name: scripts-volume
              configMap:
                name: config-server-scripts
                defaultMode: 0750
            - name: git-repo-ssh-volume
              secret:
                secretName: git-repo-ssh
                defaultMode: 0640
            - name: config-server-repo-volume
              persistentVolumeClaim:
                claimName: configserver-pvc
          restartPolicy: OnFailure

Job 详情

Name:                     config-server-worker-28664520
Namespace:                mynamespace
Selector:                 batch.kubernetes.io/controller-uid=0a368975-a112-47be-bcf5-8451a6ec90e5
Annotations:              batch.kubernetes.io/job-tracking:
Controlled By:            CronJob/config-server-worker
Parallelism:              1
Completions:              1
Completion Mode:          NonIndexed
Start Time:               Tue, 02 Jul 2024 00:00:00 +0200
Active Deadline Seconds:  120s
Pods Statuses:            0 Active (1 Ready) / 0 Succeeded / 1 Failed
Pod Template:
  Service Account:  config-server-sa
  Init Containers:
   logs-directory-creator:
    Image:      image:8.6
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
    Args:
      -c
      mkdir -p /opt/logs/app_containers/{app//mynamespace//config-server-worker//${K8S__POD_NAME}//pull-repo,audit//mynamespace//config-server-worker//${K8S__POD_NAME}//pull-repo} && chmod ugo+rwx /opt/logs/app_containers/{app//mynamespace//config-server-worker//${K8S__POD_NAME}//pull-repo,audit//mynamespace//config-server-worker//${K8S__POD_NAME}//pull-repo}
    Environment:
      K8S__POD_NAME:   (v1:metadata.name)
    Mounts:
      /opt/logs/app_containers from root-logs-directory (rw)
  Containers:
   pull-repo:
    Image:      image:1.20.10
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
    Args:
      -c
      /scripts/pull-repo.sh |& while IFS= read -r line; do printf '[%s] %s\n' "$(date '+%Y-%m-%dT%H:%M:%S.%N%:z')" "$line"; done |& tee -a /opt/logs/app/${K8S__POD_NAME}/pull-repo/pull-repo.log
    Limits:
      cpu:     50m
      memory:  300Mi
    Requests:
      cpu:      25m
      memory:   150Mi
    Liveness:   exec [ls] delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:  exec [ls] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      K8S__CONTAINER_NAME:  pull-repo
      K8S__POD_UID:          (v1:metadata.uid)
      K8S__POD_IP:           (v1:status.podIP)
      K8S__POD_NAMESPACE:    (v1:metadata.namespace)
      K8S__POD_NAME:         (v1:metadata.name)
      K8S__NODE_NAME:        (v1:spec.nodeName)
    Mounts:
      /etc/podinfo from podinfo (rw)
      /opt/app-root/src/.ssh/config from git-repo-ssh-volume (rw,path="config")
      /opt/app-root/src/.ssh/id_rsa from git-repo-ssh-volume (rw,path="id_rsa")
      /opt/app-root/src/.ssh/known_hosts from git-repo-ssh-volume (rw,path="known_hosts")
      /opt/logs/app/ from app-logs (rw)
      /opt/logs/audit/ from audit-logs (rw)
      /scripts/pull-repo.sh from scripts-volume (rw,path="pull-repo.sh")
      /tmp/git_repo_clone from config-server-repo-volume (rw)
  Volumes:
   audit-logs:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/logs/app_containers/audit/mynamespace/config-server-worker
    HostPathType:
   app-logs:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/logs/app_containers/app/mynamespace/config-server-worker
    HostPathType:
   podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
   root-logs-directory:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/logs/app_containers
    HostPathType:
   scripts-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      config-server-scripts
    Optional:  false
   git-repo-ssh-volume:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  git-repo-ssh
    Optional:    false
   config-server-repo-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  configserver-pvc
    ReadOnly:   false
Events:
  Type     Reason            Age   From            Message
  ----     ------            ----  ----            -------
  Normal   SuccessfulCreate  9h    job-controller  Created pod: config-server-worker-28664520-bt2tx
  Normal   SuccessfulDelete  9h    job-controller  Deleted pod: config-server-worker-28664520-bt2tx
  Warning  DeadlineExceeded  9h    job-controller  Job was active longer than specified deadline

问题根源分析

容器主进程未正常退出：脚本执行完成后，外层/bin/sh进程可能因管道或子进程未完全终止卡住。你的命令使用了|& tee，如果tee或后续shell子进程未收到终止信号，会导致容器主进程无法退出，kubelet无法正常杀死容器。
存储卷挂载/占用问题：Pod挂载了PVC和HostPath卷，若克隆的Git仓库文件被其他进程占用（文件句柄未释放），或存储卷出现IO阻塞、权限异常，会导致kubelet卸载卷时超时，无法完成Pod终止流程。
容器运行时与Kubelet通信故障：事件日志中的rpc error: code = DeadlineExceeded说明kubelet与containerd/cri-o通信超时，大概率是节点CPU、内存资源耗尽，导致容器运行时响应缓慢，无法及时处理杀死容器的请求。
探针配置不合理：存活和就绪探针均执行ls，容器进入终止状态后探针仍持续执行，产生大量Unhealthy警告，加剧kubelet处理负载，反而阻碍Pod终止流程。
ActiveDeadlineSeconds触发后的终止冲突：Job设置了activeDeadlineSeconds: 120，超时后会触发强制终止，但如果此时容器进程已卡住，kubelet的终止操作会因资源或进程问题超时，导致Pod停留在Terminating状态。

内容的提问来源于stack exchange，提问作者dnf

Kubernetes CronJob Pod长期处于Terminating状态问题求助

问题现象 #

相关事件日志 #

CronJob 定义 #

Job 详情 #

问题根源分析 #

问题现象

相关事件日志

CronJob 定义

Job 详情

问题根源分析