Openshift中MySQL Pod持续卡在ContainerCreating状态排查问询

我有一个用Kubernetes管理的项目，后端连接MySQL，开发环境原本运行正常。今日数据库连接突然中断，排查OpenShift时发现MySQL对应的DeploymentConfig异常，重启该Deployment后，Pod持续处于ContainerCreating状态无法启动。已删除之前故障的Pod，当前仅存在这一个无法正常启动的Pod。

Pod详情（oc describe pod mysql-1-32wk输出） #

Name:           mysql-1-32wk
Namespace:      mynamespace
Priority:       0
Node:           openshift-9kaqj-worker-b-4xpgm/10.10.0.31
Start Time:     Wed, 29 Mar 2023 12:47:48 +0200
Labels:         deployment=mysql-1
                deploymentconfig=mysql
                name=mysql
Annotations:    openshift.io/deployment-config.latest-version: 1
                openshift.io/deployment-config.name: mysql
                openshift.io/deployment.name: mysql-1
                openshift.io/scc: restricted
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicationController/mysql-1
Containers:
  mysql:
    Container ID:
    Image:          image-registry.openshift-image-registry.svc:5000/openshift/mysql@sha256:a58f67c38d33d47b46c7aa3d75e90296bfb38b73226b011afbed2e430738c45c
    Image ID:
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     300m
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   100Mi
    Liveness:   exec [/bin/sh -i -c MYSQL_PWD="$MYSQL_PASSWORD" user -u $MYSQL_USER ping] delay=30s timeout=1s period=10s #success=1 #failure=3
    Readiness:  exec [/bin/sh -i -c MYSQL_PWD="$MYSQL_PASSWORD" user -u $MYSQL_USER ping] delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      MYSQL_USER:           <set to the key 'database-user' in secret 'mysql'>           Optional: false
      MYSQL_PASSWORD:       <set to the key 'database-password' in secret 'mysql'>       Optional: false
      MYSQL_ROOT_PASSWORD:  <set to the key 'database-root-password' in secret 'mysql'>  Optional: false
      MYSQL_DATABASE:       <set to the key 'database-name' in secret 'mysql'>           Optional: false
    Mounts:
      /var/lib/mysql/data from mysql-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zks1z (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  mysql-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysql
    ReadOnly:   false
  kube-api-access-zks1z:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  13m   default-scheduler  Successfully assigned mynamespace/mysql-1-32wk to openshift-9kaqj-worker-b-4xpgm

Pod状态截图 #

可能的故障原因 #

• PersistentVolumeClaim（PVC）异常：Pod挂载的mysql-data PVC可能无法正常绑定到PV，比如PV已被删除、存储节点故障、存储类配置错误，或者PVC处于Pending状态。可通过oc get pvc mysql查看PVC状态，oc get pv检查对应PV的绑定情况。
• 镜像拉取失败：镜像仓库不可访问、镜像不存在或拉取权限不足都可能导致容器无法创建。可在节点执行podman pull image-registry.openshift-image-registry.svc:5000/openshift/mysql@sha256:a58f67c38d33d47b46c7aa3d75e90296bfb38b73226b011afbed2e430738c45c测试拉取，或查看kubelet日志journalctl -u kubelet确认错误。
• 节点资源不足：分配Pod的节点openshift-9kaqj-worker-b-4xpgm可能无足够CPU或内存资源。通过oc describe node openshift-9kaqj-worker-b-4xpgm查看节点资源使用和剩余可分配量。
• SELinux/SCC限制：使用的restricted SCC可能存在权限限制，导致无法挂载存储或启动容器。检查节点SELinux日志，或临时调整SCC权限（仅测试用途）。
• 容器运行时/kubelet故障：节点上的kubelet、containerd或docker等组件异常，无法完成容器创建流程。查看journalctl -u kubelet或systemctl status containerd确认状态。
• 网络问题：节点无法连接镜像仓库或OpenShift API，导致无法拉取镜像或获取配置。检查节点与镜像仓库的连通性，核实网络策略是否限制流量。

内容的提问来源于stack exchange，提问作者Talenel