K8s容器运行约4小时后被Topology Manager移除,附Kubelet日志求排查
K8s 容器运行4小时后被Topology Manager移除问题排查
基础环境信息
- 集群部署工具版本:
kubeadm 1.19.3 - 容器网络插件:Flannel
- 已排查排除项:底层节点无CPU、内存维度的资源过度使用情况
关联Kubelet日志摘录
Jan 07 18:35:53 vmcipind83wsfd2 kubelet[7593]: I0107 18:35:53.667179 7593 log.go:181] http: superfluous response.WriteHeader call from k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader (httplog.go:217) Jan 07 18:36:56 vmcipind83wsfd2 kubelet[7593]: I0107 18:36:56.756426 7593 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: 32346f17b258c75f77dba18796414a51b58c442c16d66a18a6f1ca32002c0bb8 Jan 07 18:37:03 vmcipind83wsfd2 kubelet[7593]: I0107 18:37:03.273923 7593 reconciler.go:196] operationExecutor.UnmountVolume started for volume "default-token-m9l42" (UniqueName: "kubernetes.io/secret/3516986a-f43b-498e-82e9-4ca11a3c3cd4-default-token-m9l42") pod "3516986a-f43b-498e-82e9-4ca11a3c3cd4" (UID: "3516986a-f43b-498e-82e9-4ca11a3c3cd4") Jan 07 18:37:03 vmcipind83wsfd2 kubelet[7593]: I0107 18:37:03.273966 7593 reconciler.go:196] operationExecutor.UnmountVolume started for volume "wta" (UniqueName: "kubernetes.io/nfs/3516986a-f43b-498e-82e9-4ca11a3c3cd4-wta-sun-trunk-wta-pv") pod "3516986a-f43b-498e-82e9-4ca11a3c3cd4" (UID: "3516986a-f43b-498e-82e9-4ca11a3c3cd4") Jan 07 18:37:03 vmcipind83wsfd2 kubelet[7593]: I0107 18:37:03.335585 7593 operation_generator.go:788] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3516986a-f43b-498e-82e9-4ca11a3c3cd4-default-token-m9l42" (OuterVolumeSpecName: "default-token-m9l42") pod "3516986a-f43b-498e-82e9-4ca11a3c3cd4" (UID: "3516986a-f43b-498e-82e9-4ca11a3c3cd4"). InnerVolumeSpecName "default-token-m9l42". PluginName "kubernetes.io/secret", VolumeGidValue "" Jan 07 18:37:03 vmcipind83wsfd2 kubelet[7593]: I0107 18:37:03.374229 7593 reconciler.go:319] Volume detached for volume "default-token-m9l42" (UniqueName: "kubernetes.io/secret/3516986a-f43b-498e-82e9-4ca11a3c3cd4-default-token-m9l42") on node "vmcipind83wsfd2" DevicePath "" Jan 07 18:37:03 vmcipind83wsfd2 kubelet[7593]: I0107 18:37:03.803952 7593 operation_generator.go:788] UnmountVolume.TearDown succeeded for volume "kubernetes.io/nfs/3516986a-f43b-498e-82e9-4ca11a3c3cd4-wta-sun-trunk-wta-pv" (OuterVolumeSpecName: "wta") pod "3516986a-f43b-498e-82e9-4ca11a3c3cd4" (UID: "3516986a-f43b-498e-82e9-4ca11a3c3cd4"). InnerVolumeSpecName "wta-sun-trunk-wta-pv". PluginName "kubernetes.io/nfs", VolumeGidValue "" Jan 07 18:37:03 vmcipind83wsfd2 kubelet[7593]: I0107 18:37:03.893440 7593 reconciler.go:319] Volume detached for volume "wta-sun-trunk-wta-pv" (UniqueName: "kubernetes.io/nfs/3516986a-f43b-498e-82e9-4ca11a3c3cd4-wta-sun-trunk-wta-pv") on node "vmcipind83wsfd2" DevicePath ""
问题排查核心方向
首先需要明确:日志中出现的[topologymanager] RemoveContainer并非Topology Manager主动驱逐容器的触发动作,这是容器销毁流程中的常规清理环节——当Kubelet判定某个容器需要终止时,会依次调用各内置组件的资源清理接口,Topology Manager的该方法仅负责释放对应容器绑定的NUMA亲和性调度状态,属于流程的结果而非诱因。
结合1.19.3版本的已知特性和“固定运行约4小时被终止”的现象,优先按以下顺序排查:
- 对齐对应Pod的生命周期配置:检查Pod是否配置了
activeDeadlineSeconds、固定超时阈值的存活探针、容器运行时的最长生命周期限制,固定时长的终止行为往往和预设的超时规则直接相关 - 核查Kubelet的驱逐配置逻辑:1.19.3版本Kubelet默认的节点资源健康检查周期为4小时,需重点检查磁盘类驱逐触发条件(包括容器日志占满分区、NFS挂载的inode耗尽、镜像存储占用超阈值),这类场景往往不会触发CPU、内存维度的资源告警,容易被“无资源过度使用”的初筛结论遗漏
- 检查Topology Manager的策略配置:若节点kubelet配置了
restricted或single-numa-node拓扑策略,当节点NUMA拓扑状态发生变化(内核NUMA平衡机制迁移了容器绑定的内存块、硬件热插拔导致NUMA节点变动),会触发绑定容器的重新调度流程 - 验证Flannel网络的长连接稳定性:1.19.3版本存在Pod心跳超时误判的已知缺陷,当Flannel链路在运行数小时后出现MTU不匹配、ARP缓存溢出等问题,会导致容器状态上报链路中断,Kubelet误判容器异常后触发驱逐
- 关注日志中
http: superfluous response.WriteHeader call异常:该异常是1.19版本Kubelet API层的已知代码缺陷,本身不会直接触发容器驱逐,但如果伴随接口响应超时,会导致Kubelet和容器运行时的状态同步偏差,间接引发容器状态误判
内容的提问来源于stack exchange,提问作者Abhishek Jain
相关产品推荐
相关产品推荐

