Helm安装prometheus-community/prometheus后Pod处于Pending状态排查

背景 #

配置跨账户指标采集到Amazon Managed Service for Prometheus（AMP），使用Helm安装prometheus-community/prometheus后，alertmanager和prometheus-server Pod持续处于Pending状态，已手动创建Persistent Volume（PV）和Persistent Volume Claim（PVC）但问题未解决。

相关配置文件 #

pv.yaml #

apiVersion: v1
kind: PersistentVolume
metadata:
  name: task-pv-volume
  labels:
    type: local
spec:
  storageClassName: gp2
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"

pvc.yaml #

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: task-pv-claim
spec:
  storageClassName: gp2
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi
  selector:
    matchLabels:
      type: local

Helm覆盖配置（values.yaml） #

serviceAccounts:
    server:
        name: "amp-iamproxy-ingest-service-account"
        annotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::${AWS-ACCOUNT-ID}:role/EKS-AMP-ServiceAccount-Role
server:
    remoteWrite:
      - url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/${WORKSPACE-ID}/api/v1/remote_write
        queue_config:
            max_samples_per_send: 1000
            max_shards: 200
            capacity: 2500
        sigv4:
            region: us-east-1
            role_arn: arn:aws:iam::${AWS-ACCOUNT-ID}:role/EKS-AMP-Central-Role
    persistentVolume:
      enabled: true
      labels:
        type: local
    emptyDir:
     sizeLimit: ""

执行命令 #

kubectl apply -f pv.yaml
kubectl apply -f pvc.yaml
helm install prometheus-for-amp prometheus-community/prometheus -n prometheus -f ./amp_ingest_override_values.yaml

相关输出 #

PVC状态 #

➜ kubectl get pvc -n prometheus                                                                                                                                                                               <aws:sandboxdemo3>
NAME                                        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
prometheus-for-amp-server                   Pending                                      gp2            19s
storage-prometheus-for-amp-alertmanager-0   Pending                                      gp2            18s
task-pv-claim                               Pending                                      gp2            28m

PVC详情（prometheus-for-amp-server） #

Name:          prometheus-for-amp-server
  Namespace:     prometheus
  StorageClass:  gp2
  Status:        Pending
  Volume:
  Labels:        app=prometheus
                 app.kubernetes.io/managed-by=Helm
                 chart=prometheus-19.7.2
                 component=server
                 heritage=Helm
                 release=prometheus-for-amp
  Annotations:   meta.helm.sh/release-name: prometheus-for-amp
                 meta.helm.sh/release-namespace: prometheus
                 volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
                 volume.kubernetes.io/selected-node: ip-192-168-31-67.ec2.internal
                 volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
  Finalizers:    [kubernetes.io/pvc-protection]
  Capacity:
  Access Modes:
  VolumeMode:    Filesystem
  Used By:       prometheus-for-amp-server-6558f799dd-4bshs
  Events:
    Type    Reason                Age                From                         Message
    ----    ------                ----               ----                         -------
    Normal  WaitForFirstConsumer  30s                persistentvolume-controller  waiting for first consumer to be created before binding
    Normal  ExternalProvisioning  13s (x3 over 29s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator

Pod状态截图 #

问题分析与解决方案 #

核心问题点 #

1. PV与PVC容量不匹配：手动创建的PV容量为1Gi，但PVC请求3Gi，导致无法绑定。
2. StorageClass冲突：EKS中gp2是EBS CSI驱动管理的动态存储类，会自动创建EBS卷；而你创建的是hostPath类型PV，两者逻辑冲突，触发了动态provisioning流程而非绑定手动PV。
3. Helm生成的PVC未关联手动PV：当前Helm配置未给server和alertmanager的PVC添加selector，无法匹配手动PV的标签。
4. hostPath PV的节点限制：hostPath类型PV仅能被同一节点的Pod使用，若Pod调度到其他节点则无法绑定。

修正步骤 #

1. 调整PV容量与StorageClass
   修改pv.yaml，将容量改为3Gi以上，并将storageClassName改为自定义名称（如local-storage）或空字符串，避免与EBS存储类混淆：

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: task-pv-volume
     labels:
       type: local
   spec:
     storageClassName: "local-storage" # 自定义存储类名称，或设为""
     capacity:
       storage: 3Gi
     accessModes:
       - ReadWriteOnce
     hostPath:
       path: "/mnt/data"
   

2. 同步PVC配置
   修改pvc.yaml，确保storageClassName与PV一致，容量匹配：

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: task-pv-claim
   spec:
     storageClassName: "local-storage"
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 3Gi
     selector:
       matchLabels:
         type: local
   

3. 更新Helm配置
   在values.yaml中为server和alertmanager的persistentVolume添加selector，并指定正确的storageClass（或空），禁用动态provisioning：

   serviceAccounts:
       server:
           name: "amp-iamproxy-ingest-service-account"
           annotations:
               eks.amazonaws.com/role-arn: arn:aws:iam::${AWS-ACCOUNT-ID}:role/EKS-AMP-ServiceAccount-Role
   server:
       remoteWrite:
         - url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/${WORKSPACE-ID}/api/v1/remote_write
           queue_config:
               max_samples_per_send: 1000
               max_shards: 200
               capacity: 2500
           sigv4:
               region: us-east-1
               role_arn: arn:aws:iam::${AWS-ACCOUNT-ID}:role/EKS-AMP-Central-Role
       persistentVolume:
         enabled: true
         storageClass: "local-storage"
         selector:
           matchLabels:
             type: local
   alertmanager:
       persistentVolume:
         enabled: true
         storageClass: "local-storage"
         selector:
           matchLabels:
             type: local
   emptyDir:
    sizeLimit: ""
   

4. 重新部署

   kubectl delete pv task-pv-volume
   kubectl delete pvc task-pv-claim
   kubectl apply -f pv.yaml
   kubectl apply -f pvc.yaml
   helm upgrade prometheus-for-amp prometheus-community/prometheus -n prometheus -f ./amp_ingest_override_values.yaml
   

若使用EKS环境，推荐直接使用EBS动态存储类（gp2/gp3），无需手动创建hostPath PV，可简化配置流程。

内容的提问来源于stack exchange，提问作者user2051904