AWS EKS Pod持续进入CrashLoopBackOff状态排查求助

我是EKS新手，正在做POC验证，想把包含Python脚本的镜像部署到EKS集群，希望基于事件触发Python脚本运行，但所有Pod都处于CrashLoopBackOff状态。

镜像Dockerfile（已发布至AWS ECR） #

FROM python:3.9-buster

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY job.py /opt/app/jobs/

# Install aws
RUN curl "https://d1vvhvl2y92vvt.cloudfront.net/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
RUN unzip awscliv2.zip
RUN ./aws/install

CMD ["bash"]

deployment.yaml配置 #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: python-job-deployment
  namespace: python-job-namespace
  labels:
    app: python-job-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: python-job-app
  template:
    metadata:
      labels:
        app: python-job-app
    spec:
      containers:
      - name: python-job
        image: xxxx.dkr.ecr.$region.amazonaws.com/python-job:latest
        ports:
        - containerPort: 80

service.yaml配置 #

apiVersion: v1
kind: Service
metadata:
  name: python-job-service
  namespace: python-job-namespace
  labels:
    app: python-job-app
spec:
  selector:
    app: python-job-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

kubectl get all -n python-job-namespace输出 #

NAME                                                    READY   STATUS             RESTARTS       AGE
pod/python-job-deployment-5998c7b884-7zwff   0/1     CrashLoopBackOff   10 (17s ago)   26m
pod/python-job-deployment-5998c7b884-b6gkp   0/1     CrashLoopBackOff   10 (19s ago)   26m
pod/python-job-deployment-5998c7b884-w5tx9   0/1     CrashLoopBackOff   10 (27s ago)   26m

NAME                                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/python-job-service   ClusterIP   x.x.x.x   <none>        80/TCP    5h21m

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/python-job-deployment   0/3     3            0           26m

NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/python-job-deployment-5998c7b884   3         3         0       26m

kubectl describe pod $podname结果 #

Name:             python-job-deployment-5998c7b884-7zwff
Namespace:        python-job-namespace
Priority:         0
Service Account:  default
Node:             ip-x-x-x-x.$region.compute.internal/x.x.x.x
Start Time:       Tue, 13 Sep 2022 15:36:41 +1000
Labels:           app=python-job-app
                  pod-template-hash=5998c7b884
Annotations:      kubernetes.io/psp: eks.privileged
Status:           Running
IP:               x.x.x.x
IPs:
  IP:           x.x.x.x
Controlled By:  ReplicaSet/python-job-deployment-5998c7b884
Containers:
  raw2std:
    Container ID:   docker://12b686344412e95968780d4bfd18ce357714f14a53b892a16bff92c49c4bef3a
    Image:          xxxxxxx.dkr.ecr.$region.amazonaws.com/python-job:latest
    Image ID:       docker-pullable://xxxxx.dkr.ecr.$region.amazonaws.com/python-job@sha256:58283c4eee275e13b8f8c531b32ed7027a667a0775f32db458c69b1b5531e5d2
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 13 Sep 2022 16:03:19 +1000
      Finished:     Tue, 13 Sep 2022 16:03:19 +1000
    Ready:          False
    Restart Count:  10
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bcqp4 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-bcqp4:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  29m                    default-scheduler  Successfully assigned python-job-namespace/python-job-deployment-5998c7b884-7zwff to ip-x-x-x-x.$region.compute.internal
  Normal   Pulled     29m                    kubelet            Successfully pulled image "xxxxxxxx.dkr.ecr.$region.amazonaws.com/python-job:latest" in 89.681294ms
  Normal   Pulled     29m                    kubelet            Successfully pulled image "xxxxxx.dkr.ecr.$region.amazonaws.com/python-job:latest" in 94.853272ms
  Normal   Pulled     29m                    kubelet            Successfully pulled image "xxxxxx.dkr.ecr.$region.amazonaws.com/python-job:latest" in 92.416417ms
  Normal   Created    29m (x4 over 29m)      kubelet            Created container raw2std
  Normal   Started    29m (x4 over 29m)      kubelet            Started container raw2std
  Normal   Pulled     29m                    kubelet            Successfully pulled image "xxxxx.dkr.ecr.$region.amazonaws.com/python-job:latest" in 111.313488ms
  Normal   Pulling    28m (x5 over 29m)      kubelet            Pulling image "xxxxx.dkr.ecr.$region.amazonaws.com/python-job:latest"
  Warning  BackOff    4m54s (x116 over 29m)  kubelet            Back-off restarting failed container

我尝试获取Pod日志，但执行kubectl logs -n python-job-namespace $podname未生成任何输出。

核心问题分析 #

从kubectl describe pod的输出可以看到，容器退出码为0（Reason: Completed），说明容器启动后执行的命令正常结束了，但Kubernetes的Deployment期望容器持续运行，所以会不断重启容器，最终进入CrashLoopBackOff状态。

你的Dockerfile中设置的CMD ["bash"]，在没有终端交互的情况下，bash会立即退出，导致容器终止。另外，你希望基于事件触发Python脚本运行，当前的Deployment模式（长期运行的Pod）并不适合这种场景。

针对性解决方案 #

1. 调整镜像的启动命令（如果要让Pod持续运行等待事件）

如果要让Pod保持运行状态以等待事件触发，可以修改Dockerfile的CMD，让容器持续运行：

# 替换原CMD，用tail保持容器存活
CMD ["tail", "-f", "/dev/null"]

或者在Deployment中覆盖启动命令：

# 在deployment.yaml的containers部分添加command
containers:
- name: python-job
  image: xxxx.dkr.ecr.$region.amazonaws.com/python-job:latest
  command: ["tail", "-f", "/dev/null"]
  ports:
  - containerPort: 80

2. 改用Kubernetes Job/CronJob（更适合事件触发的一次性任务）

如果你的Python脚本是一次性任务，基于事件触发运行，推荐使用Kubernetes Job资源，而不是Deployment：

apiVersion: batch/v1
kind: Job
metadata:
  name: python-job
  namespace: python-job-namespace
spec:
  template:
    spec:
      containers:
      - name: python-job
        image: xxxx.dkr.ecr.$region.amazonaws.com/python-job:latest
        command: ["python", "/opt/app/jobs/job.py"]
      restartPolicy: OnFailure

• 当需要触发任务时，直接创建这个Job即可；
• 如果是定时触发，改用CronJob资源。

3. 检查日志命令的正确性

你之前执行日志命令时可能打错了，正确的命令是：

kubectl logs -n python-job-namespace $podname
# 如果是查看之前终止容器的日志，加上--previous参数
kubectl logs -n python-job-namespace $podname --previous

内容的提问来源于stack exchange，提问作者Lee