EKS集群中PVC处于Pending状态的故障排查及解决咨询

在EKS 1.28集群中通过Terraform安装EBS CSI插件后，相关Pod运行正常，但创建的PVC始终处于Pending状态。

存储类信息 #

k get sc                                                                                                                                                      
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2             kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  45d
gp3 (default)   kubernetes.io/aws-ebs   Delete          Immediate              true                   19h

PVC详情 #

k describe pvc data-sonarqube-postgresql-0                                                                                                                             
Name:          data-sonarqube-postgresql-0
Namespace:     sonarqube-dce
StorageClass:  gp3
Status:        Pending
Volume:
Labels:        app.kubernetes.io/instance=sonarqube
               app.kubernetes.io/name=postgresql
               role=primary
Annotations:   volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
               volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       sonarqube-postgresql-0
Events:
  Type     Reason                Age                    From                                                                                      Message
  ----     ------                ----                   ----                                                                                      -------
  Warning  ProvisioningFailed    11m (x265 over 19h)    ebs.csi.aws.com_ebs-csi-controller-74b8b88ff9-9vngf_2fdc4610-b19f-4b1a-98a2-f3c7795528d4  failed to provision volume with StorageClass "gp3": rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Normal   ExternalProvisioning  2m5s (x4704 over 19h)  persistentvolume-controller                                                               Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal   Provisioning          54s (x281 over 19h)    ebs.csi.aws.com_ebs-csi-controller-74b8b88ff9-9vngf_2fdc4610-b19f-4b1a-98a2-f3c7795528d4  External provisioner is provisioning volume for claim "sonarqube-dce/data-sonarqube-postgresql-0"

EBS CSI控制器日志报错 #

k logs ebs-csi-controller-74b8b88ff9-9vngf -n kube-system -f                                                                                                     
Defaulted container "ebs-plugin" out of: ebs-plugin, csi-provisioner, csi-attacher, csi-snapshotter, csi-resizer, liveness-probe
I0315 11:29:02.332218       1 driver.go:83] "Driver Information" Driver="ebs.csi.aws.com" Version="v1.28.0"
I0315 11:29:02.332326       1 controller.go:92] "batching" status=true
E0316 18:55:57.576817       1 driver.go:155] "GRPC error" err=<
    rpc error: code = Internal desc = Could not create volume "pvc-20c83eb1-1afd-4a5f-8aa1-e83773edd9f2": could not create volume in EC2: RequestCanceled: request context canceled
    caused by: context deadline exceeded
 >
E0316 18:55:57.578242       1 driver.go:155] "GRPC error" err=<
    rpc error: code = Internal desc = Could not create volume "pvc-73e37c32-a8b3-4e55-bd1b-6819cf539420": could not create volume in EC2: RequestCanceled: request context canceled
    caused by: context deadline exceeded
 >
E0316 18:55:57.581023       1 driver.go:155] "GRPC error" err=<
    rpc error: code = Internal desc = Could not create volume "pvc-9a44c928-843e-4b61-a70f-d8acbc3aea06": could not create volume in EC2: RequestCanceled: request context canceled
    caused by: context canceled
 >
E0316 18:55:57.584137       1 driver.go:155] "GRPC error" err=<
    rpc error: code = Internal desc = Could not create volume "pvc-ab704102-de62-40f5-91a6-c7865c2b011e": could not create volume in EC2: RequestCanceled: request context canceled
    caused by: context deadline exceeded

• 验证IAM角色权限：确认EBS CSI控制器的IAM角色已绑定包含以下权限的策略：

  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Action": [
                  "ec2:CreateVolume",
                  "ec2:DescribeVolumes",
                  "ec2:DeleteVolume",
                  "ec2:DescribeAvailabilityZones",
                  "ec2:DescribeVolumeTypes"
              ],
              "Resource": "*"
          }
      ]
  }
  

  若用Terraform部署，检查aws_iam_role_policy_attachment资源是否正确关联该策略。

• 延长CSI控制器超时时间：修改EBS CSI控制器部署配置，添加--timeout=300s参数（默认60s）：

  kubectl edit deployment ebs-csi-controller -n kube-system
  

  在ebs-plugin容器的args中加入该参数，保存后控制器会自动滚动重启。

• 检查VPC网络连通性：

  • 若用VPC端点，确认已创建com.amazonaws.<your-region>.ec2端点，且控制器Pod所在子网安全组允许访问该端点的443端口。
  • 若用公网访问，确认控制器Pod能连接EC2公网API，节点安全组和NACL允许出站HTTPS流量。

• 排查EBS容量限制：切换存储类为gp2测试是否能创建卷，排查当前区域/可用区是否存在gp3卷容量不足的情况，必要时联系AWS支持或切换可用区。

• 重启CSI控制器Pod：临时重启修复网络连接异常：

  kubectl rollout restart deployment ebs-csi-controller -n kube-system
  

• 修改存储类绑定模式：将gp3存储类的VolumeBindingMode改为WaitForFirstConsumer，避免提前创建卷，确保卷在Pod调度的可用区生成：

  kubectl edit sc gp3
  

  修改volumeBindingMode字段为WaitForFirstConsumer。

• 预创建PV绑定PVC：手动创建EBS卷并绑定到PVC，跳过动态provisioning：

  1. 创建EBS卷并记录卷ID。
  2. 创建PV：

     apiVersion: v1
     kind: PersistentVolume
     metadata:
       name: pv-sonarqube-postgresql
     spec:
       capacity:
         storage: 10Gi # 匹配PVC需求
       accessModes:
         - ReadWriteOnce
       persistentVolumeReclaimPolicy: Delete
       storageClassName: gp3
       csi:
         driver: ebs.csi.aws.com
         volumeHandle: <your-ebs-volume-id>
     

  3. 重新创建PVC，确保storageClassName和resources.requests.storage与PV匹配。

• 配置监控告警：针对EBS CSI控制器的超时错误配置Prometheus或CloudWatch告警，及时发现问题。

• 升级CSI驱动版本：升级到最新稳定版EBS CSI驱动，修复已知的超时相关问题。

内容的提问来源于stack exchange，提问作者cloudbud