挂载EFS持久卷后Kubernetes Pod崩溃问题求助

问题背景 #

在EKS集群中使用EFS存储类部署带持久化存储的PostgreSQL，单独创建PVC能正常进入Bound状态，但将PVC挂载到Pod后，Pod无法启动，进入CrashLoopBackOff状态，无直观错误提示。

部署配置文件（deployment.yaml） #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak-deployment
  labels:
    app: keycloak
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:latest
        ports:
        - containerPort: 5432
        env:
        - name: POSTGRES_DB
          value: keycloak
        - name: POSTGRES_USER
          value: postgres
        - name: POSTGRES_PASSWORD
          value: xxxxxxxxxxxxxxx
# 去掉这部分配置后Pod能正常运行，但无持久化存储
        volumeMounts:
        - name: postgres-storage
          mountPath: /var/lib/postgresql/data
      volumes:
      - name: postgres-storage
        persistentVolumeClaim:
          claimName: postgres-keycloak-pvc
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: postgres-keycloak-pvc
  namespace: default
  labels:
    app: postgres-keycloak-pvc
spec:
  storageClassName: efs-sc
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

PVC状态 #

NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
postgres-keycloak-pvc   Bound    pvc-4fe11950-07f3-4af4-aa8f-f9e6b0fdb413   1Gi        RWX            efs-sc         <unset>                 8s

Pod Describe结果 #

Name:             keycloak-deployment-85d769f687-rfqbq
Namespace:        default
Priority:         0
Service Account:  default
Node:             xxxxxxxxxxxxxxxxxxxx
Start Time:       Wed, 06 Nov 2024 08:45:17 +0000
Labels:           app=postgres
                  pod-template-hash=85d769f687
Annotations:      <none>
Status:           Running
IP:               xxxxxxxxxx
IPs:
  IP:           xxxxxxxxxxxxx
Controlled By:  ReplicaSet/keycloak-deployment-85d769f687
Containers:
  postgres:
    Container ID:   containerd://c544f832e260640b3b0b86355764d755292f909d476c45087697de487643fb87
    Image:          postgres:latest
    Image ID:       docker.io/library/postgres@sha256:8d3be35b184e70d81e54cbcbd3df3c0b47f37d06482c0dd1c140db5dbcc6a808
    Port:           5432/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 06 Nov 2024 08:46:03 +0000
      Finished:     Wed, 06 Nov 2024 08:46:03 +0000
    Ready:          False
    Restart Count:  3
    Environment:
      POSTGRES_DB:        keycloak
      POSTGRES_USER:      postgres
      POSTGRES_PASSWORD:  xxxxxxxxxxxxxxxx
    Mounts:
      /var/lib/postgresql/data from postgres-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ngd88 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  postgres-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  postgres-keycloak-pvc
    ReadOnly:   false
  kube-api-access-ngd88:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  68s                default-scheduler  0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
  Normal   Scheduled         67s                default-scheduler  Successfully assigned default/keycloak-deployment-85d769f687-rfqbq to i-0c69be53d4d6e8dfe.eu-central-1.compute.internal
  Normal   Pulled            65s                kubelet            Successfully pulled image "postgres:latest" in 729ms (729ms including waiting). Image size: 154600919 bytes.
  Normal   Pulled            63s                kubelet            Successfully pulled image "postgres:latest" in 758ms (758ms including waiting). Image size: 154600919 bytes.
  Normal   Pulled            49s                kubelet            Successfully pulled image "postgres:latest" in 746ms (746ms including waiting). Image size: 154600919 bytes.
  Normal   Pulling           22s (x4 over 66s)  kubelet            Pulling image "postgres:latest"
  Normal   Created           21s (x4 over 65s)  kubelet            Created container postgres
  Normal   Started           21s (x4 over 65s)  kubelet            Started container postgres
  Normal   Pulled            21s                kubelet            Successfully pulled image "postgres:latest" in 738ms (738ms including waiting). Image size: 154600919 bytes.
  Warning  BackOff           7s (x6 over 63s)   kubelet            Back-off restarting failed container postgres in pod keycloak-deployment-85d769f687-rfqbq_default(cc5f184b-590f-418b-b752-94e742d565ce)

解决方案 #

1. 修复EFS挂载目录权限 #

PostgreSQL容器默认使用postgres用户（UID/GID:999）运行，而EFS挂载目录默认权限为root所有，导致PostgreSQL无法写入数据目录，这是最常见的原因。可以通过以下两种方式解决：

方式一：在Deployment中配置安全上下文

修改Deployment的容器定义，添加securityContext，确保容器运行用户和挂载目录权限匹配：

containers:
- name: postgres
  image: postgres:latest
  securityContext:
    runAsUser: 999
    runAsGroup: 999
    fsGroup: 999

fsGroup会自动将挂载卷的目录权限设置为该组ID，让postgres用户拥有读写权限。

方式二：手动修改EFS目录权限

临时启动一个挂载该PVC的Pod，手动调整目录权限：

kubectl run -it --rm efs-perm-fix --image=alpine --volume=postgres-storage:/data -- sh
chown -R 999:999 /data
exit

执行完成后，重启PostgreSQL Deployment即可。

2. 验证EFS存储类与网络配置 #

• 确认EFS存储类的provisioner配置正确（应为efs.csi.aws.com）
• 检查EFS文件系统的安全组，允许EKS节点所在安全组访问NFS端口（2049）

3. 查看容器日志获取具体错误 #

即使Pod进入CrashLoopBackOff，也可以查看上一次启动的日志，定位具体问题：

kubectl logs keycloak-deployment-85d769f687-rfqbq -c postgres --previous

日志会明确提示权限不足、目录不存在等具体错误信息。

内容的提问来源于stack exchange，提问作者Federico Chiesa