Kubernetes Pod无法挂载GCP Filestore卷 请求排查协助

我是Kubernetes新手，按教程搭建了GKE集群和GCP Filestore实例，尝试将Grafana的卷挂载到Filestore时出现超时，无法定位原因，请求协助。

Pod状态详情 #

C:\Users\ak>kubectl describe pod/grafana-7c666cff94-vkgh4
Name:           grafana-7c666cff94-vkgh4
Namespace:      bc
Priority:       0
Node:           gke-bc-gke-cluster-bc-nodepool-9496e187-zsnw/10.51.0.5
Start Time:     Fri, 02 Sep 2022 16:21:28 +0530
Labels:         app=grafana
                pod-template-hash=7c666cff94
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/grafana-7c666cff94
Containers:
  grafana:
    Container ID:
    Image:          grafana/grafana:8.4.4
    Image ID:
    Port:           3000/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        250m
      memory:     750Mi
    Liveness:     tcp-socket :3000 delay=30s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:3000/robots.txt delay=10s timeout=2s period=30s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /var/lib/grafana from fileserver (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-v7qjd (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  fileserver:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  fileserver-claim
    ReadOnly:   false
  kube-api-access-v7qjd:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                   From               Message
  ----     ------       ----                  ----               -------
  Normal   Scheduled    43m                   default-scheduler  Successfully assigned bluecopa/grafana-7c666cff94-vkgh4 to gke-bc-gke-cluster-bc-nodepool-9496e187-zsnw
  Warning  FailedMount  4m15s (x11 over 40m)  kubelet            MountVolume.SetUp failed for volume "fileserver" : mount failed: exit status 1
Mounting command: /home/kubernetes/containerized_mounter/mounter
Mounting arguments: mount -t nfs 10.168.189.130:/bc_fs /var/lib/kubelet/pods/cf44b980-7461-4c0e-a32f-673588160692/volumes/kubernetes.io~nfs/fileserver
Output: Mount failed: mount failed: exit status 32
Mounting command: chroot
Mounting arguments: [/home/kubernetes/containerized_mounter/rootfs mount -t nfs xx.xx.xx.xx:/bc_fs /var/lib/kubelet/pods/cf44b980-7461-4c0e-a32f-673588160692/volumes/kubernetes.io~nfs/fileserver]
Output: mount.nfs: Connection timed out
  Warning  FailedMount  3m16s (x12 over 37m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[fileserver], unattached volumes=[fileserver kube-api-access-v7qjd]: timed out waiting for the condition
  Warning  FailedMount  59s (x7 over 41m)     kubelet  Unable to attach or mount volumes: unmounted volumes=[fileserver], unattached volumes=[kube-api-access-v7qjd fileserver]: timed out waiting for the condition

相关配置文件 #

PV.yaml #

apiVersion: v1
kind: PersistentVolume
metadata:
  name: fileserver
  namespace: bluecopa
spec:
  capacity:
    storage: 200Gi
  accessModes:
  - ReadWriteMany
  nfs:
    path: /bc_fs
    server: xx.xx.xx.xx

PVC.yaml #

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: fileserver-claim
  namespace: bluecopa
spec:
  accessModes:
  - ReadWriteMany
  storageClassName: ""
  volumeName: fileserver
  resources:
    requests:
      storage: 100Gi

Deployment.yaml #

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
  namespace: bluecopa
spec:
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      securityContext:
        fsGroup: 472
        supplementalGroups:
          - 0
      containers:
        - name: grafana
          image: grafana/grafana:8.4.4
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http-grafana
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /robots.txt
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 2
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3000
            timeoutSeconds: 1
          resources:
            requests:
              cpu: 250m
              memory: 750Mi
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: fileserver
      volumes:
        - name: fileserver
          persistentVolumeClaim:
            claimName: fileserver-claim

排查与解决步骤 #

1. 验证网络连通性
   • 登录GKE节点，执行nc -zv xx.xx.xx.xx 2049测试NFS端口是否可达。如果不通：
     • 确认Filestore和GKE集群在同一VPC，或已配置VPC对等连接。
     • 检查Filestore的防火墙规则，允许GKE节点子网访问2049（NFS）、111（RPC）、4000-4049（挂载相关）端口。
2. 修正Filestore路径
   • Filestore的共享路径格式为/[实例名称]/[共享名称]，比如实例名bc-fs、共享名bc_fs，路径应为/bc-fs/bc_fs，核对PV中的path配置是否正确。
3. 修正PV配置
   • PV是集群级资源，不需要指定namespace，删除PV.yaml中的namespace: bluecopa字段，重新创建PV：

     kubectl delete pv fileserver
     kubectl apply -f PV.yaml
     

4. 检查节点NFS组件
   • 确认GKE节点已安装NFS客户端工具：Debian/Ubuntu节点检查nfs-common，RHEL/CentOS节点检查nfs-utils，缺失则安装。

内容的提问来源于stack exchange，提问作者CK5