无法删除部署MySQL InnoDB Cluster的Kubernetes Namespace求助
我在不同命名空间部署了MySQL Operator和MySQL InnoDB Cluster,删除Operator成功,但删除InnoDB Cluster失败。用force选项逐个删除组件后,部分成功部分失败,最后删除mycluster命名空间失败,该命名空间已处于Terminating状态5天1小时。
相关命令输出
1. 查看命名空间状态
$ kubectl get namespace mycluster mycluster Terminating 5d1h
2. 查看命名空间内资源
$ kubectl get all -n mycluster No resources found in mycluster namespace
3. 查看命名空间详细YAML
$ kubectl get namespace mycluster -o yaml apiVersion: v1 kind: Namespace metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"creationTimestamp":null,"name":"mycluster"},"spec":{},"status":{}} creationTimestamp: "2024-01-13T01:14:56Z" deletionTimestamp: "2024-01-13T02:30:18Z" labels: kubernetes.io/metadata.name: mycluster name: mycluster resourceVersion: "5011759" uid: a24383af-79b3-4e04-8ec0-fc049431d936 spec: finalizers: - kubernetes status: conditions: - lastTransitionTime: "2024-01-13T02:30:25Z" message: All resources successfully discovered reason: ResourcesDiscovered status: "False" type: NamespaceDeletionDiscoveryFailure - lastTransitionTime: "2024-01-13T02:30:25Z" message: All legacy kube types successfully parsed reason: ParsedGroupVersions status: "False" type: NamespaceDeletionGroupVersionParsingFailure - lastTransitionTime: "2024-01-13T02:30:25Z" message: All content successfully deleted, may be waiting on finalization reason: ContentDeleted status: "False" type: NamespaceDeletionContentFailure - lastTransitionTime: "2024-01-13T02:30:25Z" message: 'Some resources are remaining: innodbclusters.mysql.oracle.com has 1 resource instances' reason: SomeResourcesRemain status: "True" type: NamespaceContentRemaining - lastTransitionTime: "2024-01-13T02:30:25Z" message: 'Some content in the namespace has finalizers remaining: mysql.oracle.com/cluster in 1 resource instances' reason: SomeFinalizersRemain status: "True" type: NamespaceFinalizersRemaining phase: Terminating
4. 描述命名空间
$ kubectl describe namespace mycluster
输出内容:
Name: argocd
Labels: kubernetes.io/metadata.name=argocd
Annotations:
Status: Active
No resource quota.
No LimitRange resource.
Name: cert-manager
Labels: kubernetes.io/metadata.name=cert-manager
Annotations:
Status: Active
No resource quota.
No LimitRange resource.
Name: db-cluster
Labels: kubernetes.io/metadata.name=db-cluster
Annotations:
Status: Active
No resource quota.
No LimitRange resource.
Name: default
Labels: kubernetes.io/metadata.name=default
Annotations:
Status: Active
No resource quota.
No LimitRange resource.
Name: dev
Labels: kubernetes.io/metadata.name=dev
Annotations:
Status: Active
No resource quota.
No LimitRange resource.
Name: ingress-nginx
Labels: kubernetes.io/metadata.name=ingress-nginx
Annotations:
Status: Active
No resource quota.
No LimitRange resource.
Name: kube-node-lease
Labels: kubernetes.io/metadata.name=kube-node-lease
Annotations:
Status: Active
No resource quota.
No LimitRange resource.
Name: kube-public
Labels: kubernetes.io/metadata.name=kube-public
Annotations:
Status: Active
No resource quota.
No LimitRange resource.
Name: kube-system
Labels: kubernetes.io/metadata.name=kube-system
Annotations:
Status: Active
No resource quota.
No LimitRange resource.
Name: mycluster
Labels: kubernetes.io/metadata.name=mycluster
Annotations:
Status: Terminating
Conditions:
Type Status LastTransitionTime Reason Message
NamespaceDeletionDiscoveryFailure False Sat, 13 Jan 2024 11:30:25 +0900 ResourcesDiscovered All resources successfully discovered
NamespaceDeletionGroupVersionParsingFailure False Sat, 13 Jan 2024 11:30:25 +0900 ParsedGroupVersions All legacy kube types successfully parsed
NamespaceDeletionContentFailure False Sat, 13 Jan 2024 11:30:25 +0900 ContentDeleted All content successfully deleted, may be waiting on finalization
NamespaceContentRemaining True Sat, 13 Jan 2024 11:30:25 +0900 SomeResourcesRemain Some resources are remaining: innodbclusters.mysql.oracle.com has 1 resource instances
NamespaceFinalizersRemaining True Sat, 13 Jan 2024 11:30:25 +0900 SomeFinalizersRemain Some content in the namespace has finalizers remaining: mysql.oracle.com/cluster in 1 resource instances
No resource quota.
No LimitRange resource.
Name: mysql-operator
Labels: kubernetes.io/metadata.name=mysql-operator
Annotations:
Status: Active
No resource quota.
No LimitRange resource.
Name: redis
Labels: kubernetes.io/metadata.name=redis
Annotations:
Status: Active
No resource quota.
No LimitRange resource.
## 解决方案 从状态信息可以看出,问题根源是残留的`innodbclusters.mysql.oracle.com`自定义资源,以及它带有的`mysql.oracle.com/cluster`最终器。按以下步骤处理: ### 1. 查看并删除残留的InnoDB Cluster资源 首先列出命名空间内的所有自定义资源: ```bash kubectl get innodbclusters.mysql.oracle.com -n mycluster
如果能看到资源,直接强制删除:
kubectl delete innodbclusters.mysql.oracle.com <资源名称> -n mycluster --force --grace-period=0
2. 移除资源的最终器
如果上述删除命令失败,手动编辑资源移除最终器:
kubectl edit innodbclusters.mysql.oracle.com <资源名称> -n mycluster
在编辑器中找到metadata.finalizers字段,删除其中的mysql.oracle.com/cluster条目,保存退出。
3. 移除命名空间的最终器
如果命名空间仍卡在Terminating状态,编辑命名空间移除最终器:
kubectl edit namespace mycluster
找到spec.finalizers字段,删除kubernetes条目,保存退出。
4. 验证命名空间状态
执行以下命令确认命名空间已被删除:
kubectl get namespace mycluster
内容的提问来源于stack exchange,提问作者sa2ntjulian

