GKE挂载Filestore RWX卷Pod启动超时30分钟求助

GKE集群（版本1.30.9-gke.1127000）中，所有挂载Filestore RWX多共享卷的Pod启动耗时约30分钟，kubelet事件报如下错误：

Error syncing pod, skipping" err="unmounted volumes=[filestore-rwx-volume], unattached volumes=[], failed to process volumes=[]: context deadline exceeded"

已验证节点和Pod到Filestore实例的2049端口连通性正常，节点状态健康，挂载其他类型卷无异常。

存储配置 #

StorageClass #

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    k8s-app: gcp-filestore-csi-driver
  name: rwx-sc
parameters:
  instance-storageclass-label: rwx
  multishare: "true"
  network: prvpc
  tier: enterprise
provisioner: filestore.csi.storage.gke.io
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

PersistentVolume #

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: filestore.csi.storage.gke.io
    volume.kubernetes.io/provisioner-deletion-secret-name: ""
    volume.kubernetes.io/provisioner-deletion-secret-namespace: ""
  creationTimestamp: "2025-01-26T17:07:36Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: pv-pr
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 60Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: commerce-prodlive-assets-pvc
    namespace: commerce
    resourceVersion: "3125440"
    uid: 333cea1a-b49160c4d6e8
  csi:
    driver: filestore.csi.storage.gke.io
    volumeAttributes:
      ip: 10.xx.xx.x
      max-share-size: "1099511627776"
      storage.kubernetes.io/csiProvisionerIdentity: 123312-63xxx19-filestore.csi.storage.gke.io
      supportLockRelease: "true"
    volumeHandle: modeMultishare/enterprise-multishare-rwx-/test-k8s/europe-west1/fs-id/pv-pr
  persistentVolumeReclaimPolicy: Retain
  storageClassName: enterprise-multishare-rwx-custom
  volumeMode: Filesystem

相关日志 #

Filestore节点Pod日志（csi-driver-registrar容器） #

kubectl logs -f filestore-node-dsdsd -n kube-system
Defaulted container "csi-driver-registrar" out of: csi-driver-registrar, gcp-filestore-driver, nfs-services, filestorecsi-metrics-collector
I0309 17:00:06.274180       1 main.go:135] Version: v2.9.4-gke.27-0-gf3945690
I0309 17:00:06.274296       1 main.go:136] Running node-driver-registrar in mode=
I0309 17:00:06.274304       1 main.go:157] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0309 17:00:06.274893       1 connection.go:214] Connecting to unix:///csi/csi.sock
I0309 17:00:11.037725       1 main.go:164] Calling CSI driver to discover driver name
I0309 17:00:11.037755       1 connection.go:243] GRPC call: /csi.v1.Identity/GetPluginInfo
I0309 17:00:11.037762       1 connection.go:244] GRPC request: {}
I0309 17:00:11.041351       1 connection.go:250] GRPC response: {"name":"filestore.csi.storage.gke.io","vendor_version":"v1.6.17-gke.15"}
I0309 17:00:11.041365       1 connection.go:251] GRPC error: <nil>
I0309 17:00:11.041374       1 main.go:173] CSI driver name: "filestore.csi.storage.gke.io"
I0309 17:00:11.041408       1 node_register.go:55] Starting Registration Server at: /registration/filestore.csi.storage.gke.io-reg.sock
I0309 17:00:11.072876       1 node_register.go:64] Registration Server started at: /registration/filestore.csi.storage.gke.io-reg.sock
I0309 17:00:11.072997       1 node_register.go:88] Skipping HTTP server because endpoint is set to: ""
I0309 17:00:11.636522       1 main.go:90] Received GetInfo call: &InfoRequest{}
I0309 17:00:11.780566       1 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}

GCP Filestore CSI驱动日志 #

d8_1199_4929_a403_f386fc063f33","volume_path":"/var/lib/kubelet/pods/4facb0a5-ecb8-4b8f-839b-4141667c9bb4/volumes/kubernetes.io~csi/pvc-dcc5b9d8-1199-4929-a403-f386fc063f33/mount"}
I0418 19:02:35.517466       1 utils.go:61] GRPC response: usage:<available:23635951616 total:32212254720 used:8576303104 unit:BYTES > usage:<available:1294693 total:1966080 used:671387 unit:INODES > 
I0418 19:03:41.938784       1 utils.go:55] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0418 19:03:41.938884       1 utils.go:56] GRPC request: {}
I0418 19:03:41.938897       1 utils.go:61] GRPC response: capabilities:<rpc:<type:STAGE_UNSTAGE_VOLUME > > capabilities:<rpc:<type:GET_VOLUME_STATS > > 
I0418 19:03:41.940655       1 utils.go:55] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I0418 19:03:41.940747       1 utils.go:56] GRPC request: {"volume_id":"modeMultishare/enterprise-multishare-rwx/uat-k8s/europe-west1/fs-72382b02-20b3-4024-8206-cc6a9f95e954/pvc_dcc5b9d8_1199_4929_a403_f386fc063f33","volume_path":"/var/lib/kubelet/pods/4facb0a5-ecb8-4b8f-839b-4141667c9bb4/volumes/kubernetes.io~csi/pvc-dcc5b9d8-1199-4929-a403-f386fc063f33/mount"}
I0418 19:03:41.946455       1 utils.go:61] GRPC response: usage:<available:23635951616 total:32212254720 used:8576303104 unit:BYTES > usage:<available:1294693 total:1966080 used:671387 unit:INODES > 

I0418 19:05:22.161019       1 utils.go:55] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0418 19:05:22.161216       1 utils.go:56] GRPC request: {}
I0418 19:05:22.161232       1 utils.go:61] GRPC response: capabilities:<rpc:<type:STAGE_UNSTAGE_VOLUME > > capabilities:<rpc:<type:GET_VOLUME_STATS > > 
I0418 19:05:22.163278       1 utils.go:55] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I0418 19:05:22.163352       1 utils.go:56] GRPC request: {"volume_id":"modeMultishare/enterprise-multishare-rw/uat-k8s/europe-west1/fs-72382b02-20b3-4024-8206-cc6a9f95e954/pvc_dcc5b9d8_1199_4929_a403_f386fc063f33","volume_path":"/var/lib/kubelet/pods/4facb0a5-ecb8-4b8f-839b-4141667c9bb4/volumes/kubernetes.io~csi/pvc-dcc5b9d8-1199-4929-a403-f386fc063f33/mount"}
I0418 19:05:22.169780       1 utils.go:61] GRPC response: usage:<available:23635951616 total:32212254720 used:8576303104 unit:BYTES > usage:<available:1294693 total:1966080 used:671387 unit:INODES > 

排查建议 #

• CSI驱动版本兼容性检查：当前Filestore CSI驱动版本为v1.6.17-gke.15，确认其与GKE 1.30.9-gke.1127000版本的兼容性，必要时升级到对应GKE版本推荐的驱动版本。
• 多共享卷锁机制排查：启用multishare: "true"后，Filestore依赖锁机制管理共享访问，检查Filestore实例是否存在锁资源竞争，或节点上的锁代理组件运行状态。
• kubelet挂载超时调整：kubelet默认挂载超时可能不足以处理Filestore多共享卷的挂载流程，可调整kubelet相关超时参数，或查看kubelet日志中挂载步骤的详细耗时。
• Filestore实例负载监控：通过GCP控制台查看Filestore实例的CPU、IOPS负载，若实例处于高负载状态会导致挂载响应延迟。
• 节点NFS客户端配置检查：确认节点NFS客户端版本与Filestore兼容，检查NFS挂载参数（如硬挂载超时）是否合理。
• PVC/PV绑定状态验证：检查PVCcommerce-prodlive-assets-pvc的绑定状态是否正常，排除延迟绑定导致的挂载等待问题。

内容的提问来源于stack exchange，提问作者saurabh umathe