卸载Longhorn时Volume卡在deleting状态的问题求助

在3台RHEL 8服务器搭建的k3s集群中，执行helm uninstall longhorn -n longhorn-system卸载Longhorn后，所有Longhorn Pod、PVC等资源都已删除，但有一个Volume始终卡在deleting状态。该Volume的详细信息如下：

Name:         pvc-f1df1bf8-96f4-4b28-a14d-2b20809610df
Namespace:    longhorn-system
Labels:       longhornvolume=pvc-f1df1bf8-96f4-4b28-a14d-2b20809610df
              recurring-job-group.longhorn.io/default=enabled
              setting.longhorn.io/remove-snapshots-during-filesystem-trim=ignored
              setting.longhorn.io/replica-auto-balance=ignored
              setting.longhorn.io/snapshot-data-integrity=ignored
Annotations:  <none>
API Version:  longhorn.io/v1beta2
Kind:         Volume
Metadata:
  Creation Timestamp:             2023-08-21T07:31:56Z
  Deletion Grace Period Seconds:  0
  Deletion Timestamp:             2023-08-24T09:32:05Z
  Finalizers:
    longhorn.io
  Generation:        214
  Resource Version:  7787140
  UID:               6ffb214d-8ed7-4b7b-910e-a2936b764223
Spec:
  Standby:           false
  Access Mode:       rwo
  Backing Image:
  Base Image:
  Data Locality:     disabled
  Data Source:
  Disable Frontend:  false
  Disk Selector:
  Encrypted:          false
  Engine Image:       longhornio/longhorn-engine:v1.4.1
  From Backup:
  Frontend:           blockdev
  Last Attached By:
  Migratable:         false
  Migration Node ID:
  Node ID:
  Node Selector:
  Number Of Replicas:  3
  Recurring Jobs:
  Replica Auto Balance:           ignored
  Restore Volume Recurring Job:   ignored
  Revision Counter Disabled:      false
  Size:                           4294967296
  Snapshot Data Integrity:        ignored
  Stale Replica Timeout:          30
  Unmap Mark Snap Chain Removed:  ignored
Status:
  Actual Size:  0
  Clone Status:
    Snapshot:
    Source Volume:
    State:
  Conditions:
    Last Probe Time:
    Last Transition Time:  2023-08-21T07:31:57Z
    Message:
    Reason:
    Status:                False
    Type:                  toomanysnapshots
    Last Probe Time:
    Last Transition Time:  2023-08-21T07:31:57Z
    Message:
    Reason:
    Status:                True
    Type:                  scheduled
    Last Probe Time:
    Last Transition Time:  2023-08-21T07:31:57Z
    Message:
    Reason:
    Status:                False
    Type:                  restore
  Current Image:           longhornio/longhorn-engine:v1.4.1
  Current Node ID:
  Expansion Required:      false
  Frontend Disabled:       false
  Is Standby:              false
  Kubernetes Status:
    Last PVC Ref At:  2023-08-24T09:32:04Z
    Last Pod Ref At:  2023-08-24T09:24:48Z
    Namespace:        backend
    Pv Name:
    Pv Status:
    Pvc Name:         pvc-longhorn-db
    Workloads Status:
      Pod Name:          wb-database-deployment-8685cbdcfc-2dfs2
      Pod Status:        Failed
      Workload Name:     wb-database-deployment-8685cbdcfc
      Workload Type:     ReplicaSet
  Last Backup:
  Last Backup At:
  Last Degraded At:
  Owner ID:              node3
  Pending Node ID:
  Remount Requested At:  2023-08-24T09:23:55Z
  Restore Initiated:     false
  Restore Required:      false
  Robustness:            unknown
  Share Endpoint:
  Share State:
  State:                 deleting
Events:                  <none>

已尝试移除finalizers但问题未解决，求问该Volume无法正常卸载的原因。

核心原因分析 #

• Longhorn控制组件缺失：卸载Longhorn后，负责处理Volume删除逻辑的manager、controller等组件已被移除，导致Volume的longhorn.io finalizer无法被正常处理；即使手动移除finalizer，K8s层面可能仍残留未清理的关联依赖。
• Failed Pod残留引用：从Volume状态可见，backend命名空间下有一个Failed状态的Pod（wb-database-deployment-8685cbdcfc-2dfs2）仍关联该Volume，残留的挂载绑定或内部引用会阻止Volume彻底删除。
• 节点本地资源残留：该Volume的Owner ID为node3，可能在node3节点上还存在对应的Longhorn卷数据目录、块设备等物理资源，这些残留会导致K8s资源无法被完全清理。
• Volume状态异常：Volume的Robustness为unknown，说明卸载Longhorn前该Volume状态已异常，未完成数据同步或前置清理流程，导致删除卡住。

分步解决建议 #

1. 临时恢复Longhorn处理删除
   重新安装同版本（v1.4.1）的Longhorn，让控制组件重新接管Volume：

   helm install longhorn longhorn/longhorn -n longhorn-system --version 1.4.1
   

   待组件启动后，通过Longhorn UI或kubectl命令删除该Volume，完成后再重新卸载Longhorn。

2. 清理关联的Failed Pod
   强制删除残留的Failed Pod，解除Volume引用：

   kubectl delete pod wb-database-deployment-8685cbdcfc-2dfs2 -n backend --force --grace-period=0
   

   之后再尝试删除Volume资源。

3. 手动清理节点本地残留
   登录node3节点，找到Longhorn默认数据目录（/var/lib/longhorn/volumes/），删除对应Volume的目录：

   rm -rf /var/lib/longhorn/volumes/pvc-f1df1bf8-96f4-4b28-a14d-2b20809610df
   

   同时检查是否有残留的块设备（可通过lsblk命令排查），清理后回到集群中重新删除Volume。

4. 强制清理K8s资源（谨慎操作）
   若以上方法无效，可强制清理Volume的finalizer并删除资源：

   kubectl patch volume pvc-f1df1bf8-96f4-4b28-a14d-2b20809610df -n longhorn-system -p '{"metadata":{"finalizers":null}}' --type=merge
   kubectl delete volume pvc-f1df1bf8-96f4-4b28-a14d-2b20809610df -n longhorn-system --force --grace-period=0
   

内容的提问来源于stack exchange，提问作者Oberwalder Sven