重启Minio-3 Pod时卷已绑定旧Pod问题求助

执行kubectl delete -n minio pod minio-3重启Pod后，新Pod一直卡在ContainerCreating状态，kubelet报错**"Volume is already attached by pod minio/minio-3"**，卷被旧Pod实例的残留绑定占用。

排查信息 #

1. Pod状态 #

[root@control01 ~]# kubectl get pod -n minio 
NAME      READY   STATUS              RESTARTS   AGE
minio-0   0/1     ContainerCreating   0          62m
minio-1   1/1     Running             0          128m
minio-2   1/1     Running             1          6d7h
minio-3   0/1     ContainerCreating   0          96m

2. kubelet日志（/var/log/messages） #

Sep 13 16:48:10 control12 kubelet: E0913 16:48:10.189343   40141 nestedpendingoperations.go:270] Operation for "\"flexvolume-ceph.rook.io/rook-ceph/e51e0fbd-f09f-430a-8b47-2ca1dbdfdc2e-pvc-06a22ce4-cbbb-4cd7-82c5-d7bf9755fbd9\" (\"e51e0fbd-f09f-430a-8b47-2ca1dbdfdc2e\")" failed. No retries permitted until 2022-09-13 16:50:12.189281072 +0800 CST m=+5554.416909175 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume \"pvc-06a22ce4-cbbb-4cd7-82c5-d7bf9755fbd9\" (UniqueName: \"flexvolume-ceph.rook.io/rook-ceph/e51e0fbd-f09f-430a-8b47-2ca1dbdfdc2e-pvc-06a22ce4-cbbb-4cd7-82c5-d7bf9755fbd9\") pod \"minio-3\" (UID: \"e51e0fbd-f09f-430a-8b47-2ca1dbdfdc2e\") : mount command failed, status: Failure, reason: Rook: Mount volume failed: failed to attach volume pvc-06a22ce4-cbbb-4cd7-82c5-d7bf9755fbd9 for pod minio/minio-3. Volume is already attached by pod minio/minio-3. Status Pending"

3. Pod详情（kubectl describe pod minio-3） #

Name:           minio-3
Namespace:      minio
Priority:       0
Node:           control12/192.168.1.112
Start Time:     Tue, 13 Sep 2022 15:18:28 +0800
Labels:         app=minio
                controller-revision-hash=minio-95c8c444c
                statefulset.kubernetes.io/pod-name=minio-3
Annotations:    <none>
Status:         Pending
IP:              
IPs:            <none>
Controlled By:  StatefulSet/minio
Containers:
  minio:
    Container ID:  
    Image:         minio/minio:RELEASE.2021-04-06T23-11-00Z
    Image ID:      
    Port:          9000/TCP
    Host Port:     0/TCP
    Args:
      server
      http://minio-{0...3}.minio.minio.svc.cluster.local/minio/data
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
   ...
   ...
Events:
  Type     Reason       Age                  From                Message
  ----     ------       ----                 ----                -------
  Warning  FailedMount  52m (x13 over 93m)   kubelet, control12  Unable to attach or mount volumes: unmounted volumes=[minio-data], unattached volumes=[minio-data tz-config default-token-np5x5]: timed out waiting for the condition
  Warning  FailedMount  11m (x50 over 97m)   kubelet, control12  MountVolume.SetUp failed for volume "pvc-06a22ce4-cbbb-4cd7-82c5-d7bf9755fbd9" : mount command failed, status: Failure, reason: Rook: Mount volume failed: failed to attach volume pvc-06a22ce4-cbbb-4cd7-82c5-d7bf9755fbd9 for pod minio/minio-3. Volume is already attached by pod minio/minio-3. Status Pending
  Warning  FailedMount  6m36s (x7 over 95m)  kubelet, control12  Unable to attach or mount volumes: unmounted volumes=[minio-data], unattached volumes=[tz-config default-token-np5x5 minio-data]: timed out waiting for the condition
  Warning  FailedMount  2m1s (x9 over 86m)   kubelet, control12  Unable to attach or mount volumes: unmounted volumes=[minio-data], unattached volumes=[default-token-np5x5 minio-data tz-config]: timed out waiting for the condition

解决方案 #

步骤1：确认卷的绑定状态 #

先获取PVC对应的PV信息，定位卷的唯一标识符：

kubectl get pvc -n minio
kubectl get pv pvc-06a22ce4-cbbb-4cd7-82c5-d7bf9755fbd9 -o yaml

在PV的YAML输出中，找到卷的ID（flexvolume格式通常为pvc-xxxxxx，CSI格式为csi-xxxxxx），同时检查status.phase和绑定的Pod信息。

步骤2：清理Rook/Ceph中的残留绑定 #

由于使用Rook管理Ceph存储，直接在Ceph集群内解绑卷：

1. 进入Rook工具Pod：

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash

2. 查看卷的挂载状态：

rbd status <VOLUME_ID>

（替换<VOLUME_ID>为步骤1中获取的卷ID）
3. 若卷仍被旧Pod绑定，强制解绑：

rbd unmap <VOLUME_ID>

步骤3：清理Kubernetes中的绑定残留记录 #

删除Kubernetes API中残留的VolumeAttachment资源：

1. 查找关联目标PVC的VolumeAttachment：

kubectl get volumeattachment | grep pvc-06a22ce4-cbbb-4cd7-82c5-d7bf9755fbd9

2. 删除该资源：

kubectl delete volumeattachment <VOLUME_ATTACHMENT_NAME>

（替换<VOLUME_ATTACHMENT_NAME>为上一步查到的资源名称）

步骤4：重新启动Pod #

删除当前卡住的Pod，让StatefulSet重新创建并挂载卷：

kubectl delete pod minio-3 -n minio

预防措施 #

• 避免直接kubectl delete pod强制删除StatefulSet Pod，优先使用kubectl rollout restart statefulset minio -n minio进行滚动重启，让Kubernetes正常处理卷的解绑流程。
• 确保Rook/Ceph集群版本与Kubernetes版本兼容，避免驱动bug导致的卷绑定残留。

内容的提问来源于stack exchange，提问作者willyao