GKE集群standard-rwo持久化磁盘无法挂载问题求助

云上老周

2026-7-14

GKE集群standard-rwo类型PV挂载失败排查

问题背景

GKE集群升级至1.25.10-gke.2700版本后，所有standard-rwo类型的持久化磁盘（包括新创建的）均无法挂载。尝试卸载并重新安装Helm Chart、手动创建PV均无效。

现象示例（Prometheus栈动态PV）

PV状态

kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                                                                                    STORAGECLASS   REASON   AGE
pvc-835b43a5-56b3-4811-8641-a6484d82af94   5Gi        RWO            Delete           Bound    kube-monitoring/prometheus-grafana                                                                                       standard-rwo            10m
pvc-cf74cd43-bb11-458c-90f7-2cfe2fcf8f51   30Gi       RWO            Delete           Bound    kube-monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0   standard-rwo            10m

PVC状态

kubectl get pvc -n kube-monitoring
NAME                                                                                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
prometheus-grafana                                                                                       Bound    pvc-835b43a5-56b3-4811-8641-a6484d82af94   5Gi        RWO            standard-rwo   12m
prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0   Bound    pvc-cf74cd43-bb11-458c-90f7-2cfe2fcf8f51   30Gi       RWO            standard-rwo   11m

关键事件日志

Events:
  Type     Reason                  Age                  From                     Message
  ----     ------                  ----                 ----                     -------
  Normal   Scheduled               4m32s                default-scheduler        Successfully assigned kube-monitoring/prometheus-grafana-f8bc76bb7-55lgt to gke-dev-wp-hosting-manage-pool-nat-5e3cde5c-19k6
  Normal   SuccessfulAttachVolume  4m27s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-835b43a5-56b3-4811-8641-a6484d82af94"
  Warning  FailedMount             14s (x2 over 2m29s)  kubelet                  Unable to attach or mount volumes: unmounted volumes=[storage], unattached volumes=[sc-datasources-volume config sc-dashboard-provider storage kube-api-access-6hgdb sc-dashboard-volume]: timed out waiting for the condition
  Warning  FailedMount             7s (x10 over 4m23s)  kubelet                  MountVolume.MountDevice failed for volume "pvc-835b43a5-56b3-4811-8641-a6484d82af94" : rpc error: code = Internal desc = Failed to format and mount device from ("/dev/disk/by-id/google-pvc-835b43a5-56b3-4811-8641-a6484d82af94") to ("/var/lib/kubelet/plugins/kubernetes.io/csi/pd.csi.storage.gke.io/4940624ccce6daf2ad2958c9e49dc6e1a6249274233f0c2ab90018bee6468ea3/globalmount") with fstype ("ext4") and options ([]): mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t ext4 -o defaults /dev/disk/by-id/google-pvc-835b43a5-56b3-4811-8641-a6484d82af94 /var/lib/kubelet/plugins/kubernetes.io/csi/pd.csi.storage.gke.io/4940624ccce6daf2ad2958c9e49dc6e1a6249274233f0c2ab90018bee6468ea3/globalmount
Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pd.csi.storage.gke.io/4940624ccce6daf2ad2958c9e49dc6e1a6249274233f0c2ab90018bee6468ea3/globalmount: cannot mount /dev/sdb read-only.

PV YAML配置

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: pd.csi.storage.gke.io
    volume.kubernetes.io/provisioner-deletion-secret-name: ""
    volume.kubernetes.io/provisioner-deletion-secret-namespace: ""
  creationTimestamp: "2023-08-03T07:50:06Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/pd-csi-storage-gke-io
  name: pvc-835b43a5-56b3-4811-8641-a6484d82af94
  resourceVersion: "404236306"
  uid: 07fc6f01-8449-4fef-ad2a-dcbd04a46831
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 5Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: prometheus-grafana
    namespace: kube-monitoring
    resourceVersion: "404236229"
    uid: 835b43a5-56b3-4811-8641-a6484d82af94
  csi:
    driver: pd.csi.storage.gke.io
    fsType: ext4
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: 1691044660229-8081-pd.csi.storage.gke.io
    volumeHandle: projects/xxx/zones/us-central1-f/disks/pvc-835b43a5-56b3-4811-8641-a6484d82af94
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.gke.io/zone
          operator: In
          values:
          - us-central1-f
  persistentVolumeReclaimPolicy: Delete
  storageClassName: standard-rwo
  volumeMode: Filesystem
status:
  phase: Bound

PVC YAML配置

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    meta.helm.sh/release-name: prometheus
    meta.helm.sh/release-namespace: kube-monitoring
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
    volume.kubernetes.io/selected-node: gke-dev-wp-hosting-manage-pool-nat-5e3cde5c-19k6
    volume.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
  creationTimestamp: "2023-08-03T07:49:59Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana
    app.kubernetes.io/version: 10.0.2
    helm.sh/chart: grafana-6.58.4
  name: prometheus-grafana
  namespace: kube-monitoring
  resourceVersion: "404236299"
  uid: 835b43a5-56b3-4811-8641-a6484d82af94
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: standard-rwo
  volumeMode: Filesystem
  volumeName: pvc-835b43a5-56b3-4811-8641-a6484d82af94
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 5Gi
  phase: Bound

问题分析与解决步骤

核心问题定位

从事件日志中的cannot mount /dev/sdb read-only可以明确：磁盘被以只读模式附加到节点，但ReadWriteOnce模式需要读写权限，导致挂载失败。

排查与修复步骤

验证磁盘附加模式
使用gcloud命令检查目标磁盘的附加状态，确认是否为只读：
```
gcloud compute disks describe pvc-835b43a5-56b3-4811-8641-a6484d82af94 --zone us-central1-f
```
在输出中查找users字段，若包含ro标识（如projects/xxx/zones/us-central1-f/instances/gke-dev-wp-hosting-manage-pool-nat-5e3cde5c-19k6:ro），说明磁盘是只读附加。

重新以读写模式附加磁盘

先删除对应的Pod和PVC，确保磁盘被自动卸载：

kubectl delete pod prometheus-grafana-f8bc76bb7-55lgt -n kube-monitoring
kubectl delete pvc prometheus-grafana -n kube-monitoring

手动 detach 磁盘：

gcloud compute instances detach-disk gke-dev-wp-hosting-manage-pool-nat-5e3cde5c-19k6 --disk pvc-835b43a5-56b3-4811-8641-a6484d82af94 --zone us-central1-f

重新创建PVC，让CSI驱动以读写模式附加磁盘。

检查PD CSI驱动版本
GKE集群的PD CSI驱动可能存在兼容性问题，查看当前驱动版本：
```
kubectl get pods -n kube-system | grep pd-csi
```
确保驱动版本与GKE 1.25.10-gke.2700兼容。若版本过旧，升级驱动：
```
gcloud container clusters update YOUR_CLUSTER_NAME --update-addons=GcePersistentDiskCsiDriver=ENABLED --zone YOUR_ZONE
```

检查节点状态与组件日志

查看节点是否有硬件或系统层面的磁盘控制器故障：

gcloud compute instances get-serial-port-output gke-dev-wp-hosting-manage-pool-nat-5e3cde5c-19k6 --zone us-central1-f

查看kubelet和CSI节点组件日志，排查挂载流程中的异常：

kubectl logs -n kube-system gke-pd-csi-node-xxx -c node-driver-registrar
kubectl logs -n kube-system gke-pd-csi-node-xxx -c pd-csi-driver

验证存储类配置
检查standard-rwo存储类是否被意外修改，确保没有添加readonly相关参数：
```
kubectl get storageclass standard-rwo -o yaml
```
正常配置应包含provisioner: pd.csi.storage.gke.io，且无强制只读的参数。
重启节点（临时故障排查）
若上述步骤无效，尝试重启故障节点，排除临时系统故障：
```
gcloud compute instances restart gke-dev-wp-hosting-manage-pool-nat-5e3cde5c-19k6 --zone us-central1-f
```

内容的提问来源于stack exchange，提问作者Vitaliy Isarev

GKE集群standard-rwo持久化磁盘无法挂载问题求助

问题背景 #

现象示例（Prometheus栈动态PV） #

PV状态 #

PVC状态 #

关键事件日志 #

PV YAML配置 #

PVC YAML配置 #

问题分析与解决步骤 #

核心问题定位 #

排查与修复步骤 #