Kubernetes中MongoDB Pod处于CrashLoopBackOff状态求助

问题背景 #

学习Kubernetes时部署MongoDB实例，出现部分Pod处于CrashLoopBackOff状态，部分处于ImagePullBackOff状态，容器启动后立即终止，退出码为1。

我的配置文件 #

Deployment配置 #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-deployment
  labels:
    app: mongodb
spec:
  replicas: 2
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      containers:
      - name: mongodb
        image: mongo
        ports:
        - containerPort: 27017
        env:
        - name: MONGO_INITDB_ROOT_USERNAME
          valueFrom:
            secretKeyRef: 
              name: mongodb-secret
              key: mongo-root-username
        - name: MONGO_INITIDB_ROOT_PASSWORD
          valueFrom:
            secretKeyRef: 
              name: mongodb-secret
              key: mongo-root-password

Secret配置 #

apiVersion: v1
kind: Secret
metadata:
  name: mongodb-secret
type: 0paque
data:
  mongo-root-username: dXNlcm5hbWU=
  mongo-root-password: cGFzc3dvcmQ=

异常状态信息 #

执行kubectl get all结果：

kubectl get all
NAME                                     READY   STATUS             RESTARTS      AGE  
pod/mongodb-deployment-6ddd5fb89-h9rjz   0/1     ImagePullBackOff   0             7m25s
pod/mongodb-deployment-6ddd5fb89-wvz6p   0/1     ImagePullBackOff   0             7m25s
pod/mongodb-deployment-f7df49f67-2gp4x   0/1     CrashLoopBackOff   5 (43s ago)   3m54s
pod/nginx-deployment-78cc6468fb-22wz5    1/1     Running            0             49m
pod/nginx-deployment-78cc6468fb-6hxq8    1/1     Running            0             49m

NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/kubernetes      ClusterIP   10.96.0.1       <none>        443/TCP   68m
service/nginx-service   ClusterIP   10.110.136.45   <none>        80/TCP    34m

NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mongodb-deployment   0/2     1            0           7m25s
deployment.apps/nginx-deployment     2/2     2            2           49m

NAME                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/mongodb-deployment-6ddd5fb89   2         2         0       7m25s
replicaset.apps/mongodb-deployment-f7df49f67   1         1         0       3m54s
replicaset.apps/nginx-deployment-78cc6468fb    2         2         2       49m

CrashLoopBackOff状态Pod详情：

kubectl describe pod/mongodb-deployment-f7df49f67-2gp4x           
Name:         mongodb-deployment-f7df49f67-2gp4x
Namespace:    default
Priority:     0
Node:         minikube/192.168.49.2
Start Time:   Thu, 06 Oct 2022 13:10:09 -0500
Labels:       app=mongodb
              pod-template-hash=f7df49f67
Annotations:  <none>
Status:       Running
IP:           172.17.0.7
IPs:
  IP:           172.17.0.7
Controlled By:  ReplicaSet/mongodb-deployment-f7df49f67
Containers:
  mongodb:
    Container ID:   docker://73f6707f1fc2b5ae690cf3518e35ab05d258a3f209b106c014310b0e38c05f00
    Image:          mongo
    Image ID:       docker-pullable://mongo@sha256:2ca8fb22c9522b49fd1f5490dee3e7026a4331b9f904d5acf10a9638c1d1539d
    Port:           27017/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 06 Oct 2022 13:11:47 -0500
      Finished:     Thu, 06 Oct 2022 13:11:47 -0500
    Ready:          False
    Restart Count:  4
    Environment:
      MONGO_INITDB_ROOT_USERNAME:   <set to the key 'mongo-root-username' in secret 'mongodb-secret'>  Optional: false
      MONGO_INITIDB_ROOT_PASSWORD:  <set to the key 'mongo-root-password' in secret 'mongodb-secret'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gpmw4 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  2m17s                default-scheduler  Successfully assigned default/mongodb-deployment-f7df49f67-2gp4x to minikube
  Normal   Pulled     2m13s                kubelet            Successfully pulled image "mongo" in 2.4540099s
  Normal   Pulled     2m11s                kubelet            Successfully pulled image "mongo" in 2.0397836s
  Normal   Pulled     113s                 kubelet            Successfully pulled image "mongo" in 2.5676941s
  Normal   Created    84s (x4 over 2m13s)  kubelet            Created container mongodb
  Normal   Started    84s (x4 over 2m13s)  kubelet            Started container mongodb
  Normal   Pulled     84s                  kubelet            Successfully pulled image "mongo" in 2.2913276s
  Warning  BackOff    53s (x7 over 2m10s)  kubelet            Back-off restarting failed container
  Normal   Pulling    42s (x5 over 2m16s)  kubelet            Pulling image "mongo"
  Normal   Pulled     39s                  kubelet            Successfully pulled image "mongo" in 2.3732048s

问题排查与修复 #

1. CrashLoopBackOff 核心问题 #

从配置中可以定位两个致命错误：

• 环境变量名拼写错误：Deployment中密码变量名写成了MONGO_INITIDB_ROOT_PASSWORD，正确的MongoDB初始化变量名应为MONGO_INITDB_ROOT_PASSWORD（注意是INITDB而非INITIDB）。错误的变量名导致MongoDB无法正确初始化，容器启动后直接退出。
• Secret类型错误：Secret的type字段写成了数字0paque，正确类型应为大写字母开头的Opaque，虽然Kubernetes可能兼容错误类型，但不符合规范。

修复后的配置

Deployment配置

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-deployment
  labels:
    app: mongodb
spec:
  replicas: 2
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      containers:
      - name: mongodb
        image: mongo
        ports:
        - containerPort: 27017
        env:
        - name: MONGO_INITDB_ROOT_USERNAME
          valueFrom:
            secretKeyRef: 
              name: mongodb-secret
              key: mongo-root-username
        - name: MONGO_INITDB_ROOT_PASSWORD
          valueFrom:
            secretKeyRef: 
              name: mongodb-secret
              key: mongo-root-password

Secret配置

apiVersion: v1
kind: Secret
metadata:
  name: mongodb-secret
type: Opaque
data:
  mongo-root-username: dXNlcm5hbWU=
  mongo-root-password: cGFzc3dvcmQ=

2. ImagePullBackOff 问题处理 #

该状态通常是节点拉取镜像时网络不稳定导致，可执行以下操作：

• 手动在Minikube节点拉取镜像：minikube ssh docker pull mongo
• 删除异常Pod，让Deployment重新创建：kubectl delete pod mongodb-deployment-6ddd5fb89-h9rjz mongodb-deployment-6ddd5fb89-wvz6p

验证修复 #

应用修复后的配置：

kubectl apply -f mongodb-secret.yaml
kubectl apply -f mongodb-deployment.yaml

等待数分钟后执行kubectl get pods，确认所有MongoDB Pod进入Running状态。

内容的提问来源于stack exchange，提问作者sebsmgzz