Kubernetes 1.25容器随机启停问题排查求助

问题场景 #

我安装了Kubernetes 1.25版本，执行kubeadm config images pull后拉取到以下镜像：

> crictl images
IMAGE                                     TAG                 IMAGE ID            SIZE
k8s.gcr.io/pause                          3.6                 6270bb605e12e       302kB
registry.k8s.io/coredns/coredns           v1.9.3              5185b96f0becf       14.8MB
registry.k8s.io/etcd                      3.5.4-0             a8a176a5d5d69       102MB
registry.k8s.io/kube-apiserver            v1.25.1             b09a3dc327be2       34.2MB
registry.k8s.io/kube-controller-manager   v1.25.1             4d904f9df20a0       31.3MB
registry.k8s.io/kube-controller-manager   v1.25.2             dbfceb93c69b6       31.3MB
registry.k8s.io/kube-proxy                v1.25.1             3ed3cb68c861e       20.3MB
registry.k8s.io/kube-proxy                v1.25.2             1c7d8c51823b5       20.3MB
registry.k8s.io/kube-scheduler            v1.25.1             4df0903523686       15.8MB
registry.k8s.io/pause                     3.8                 4873874c08efc       311kB

通过crictl ps输出可见，部分容器（如kube-proxy）会随机停止运行，之后自行恢复：

root@debian:~# crictl ps 
CONTAINER           IMAGE               CREATED              STATE               NAME                      ATTEMPT             POD ID              POD
57ea1fbf9a290       a8a176a5d5d69       42 seconds ago       Running             etcd                      1876                d29bb8c1c84d3       etcd-debian
9ae0603c3489b       3ed3cb68c861e       59 seconds ago       Running             kube-proxy                117                 e259b18a6b09b       kube-proxy-8t2mz
9e5d71a57430a       4df0903523686       About a minute ago   Running             kube-scheduler            132                 07b6133ee8f16       kube-scheduler-debian
6fbf351ab72d3       4d904f9df20a0       About a minute ago   Running             kube-controller-manager   124                 b25394a8165e9       kube-controller-manager-debian
627fdc8f37984       b09a3dc327be2       2 minutes ago        Running             kube-apiserver            123                 b510c0c32211d       kube-apiserver-debian
root@debian:~# crictl ps 
CONTAINER           IMAGE               CREATED              STATE               NAME                      ATTEMPT             POD ID              POD
57ea1fbf9a290       a8a176a5d5d69       50 seconds ago       Running             etcd                      1876                d29bb8c1c84d3       etcd-debian
9ae0603c3489b       3ed3cb68c861e       About a minute ago   Running             kube-proxy                117                 e259b18a6b09b       kube-proxy-8t2mz
9e5d71a57430a       4df0903523686       About a minute ago   Running             kube-scheduler            132                 07b6133ee8f16       kube-scheduler-debian
6fbf351ab72d3       4d904f9df20a0       About a minute ago   Running             kube-controller-manager   124                 b25394a8165e9       kube-controller-manager-debian
627fdc8f37984       b09a3dc327be2       2 minutes ago        Running             kube-apiserver            123                 b510c0c32211d       kube-apiserver-debian
root@debian:~# crictl ps 
CONTAINER           IMAGE               CREATED              STATE               NAME                      ATTEMPT             POD ID              POD
57ea1fbf9a290       a8a176a5d5d69       51 seconds ago       Running             etcd                      1876                d29bb8c1c84d3       etcd-debian
9e5d71a57430a       4df0903523686       About a minute ago   Running             kube-scheduler            132                 07b6133ee8f16       kube-scheduler-debian
6fbf351ab72d3       4d904f9df20a0       About a minute ago   Running             kube-controller-manager   124                 b25394a8165e9       kube-controller-manager-debian
627fdc8f37984       b09a3dc327be2       2 minutes ago        Running             kube-apiserver            123                 b510c0c32211d       kube-apiserver-debian
root@debian:~# crictl ps 
CONTAINER           IMAGE               CREATED              STATE               NAME                      ATTEMPT             POD ID              POD
57ea1fbf9a290       a8a176a5d5d69       52 seconds ago       Running             etcd                      1876                d29bb8c1c84d3       etcd-debian
9e5d71a57430a       4df0903523686       About a minute ago   Running             kube-scheduler            132                 07b6133ee8f16       kube-scheduler-debian
6fbf351ab72d3       4d904f9df20a0       About a minute ago   Running             kube-controller-manager   124                 b25394a8165e9       kube-controller-manager-debian
627fdc8f37984       b09a3dc327be2       2 minutes ago        Running             kube-apiserver            123                 b510c0c32211d       kube-apiserver-debian
root@debian:~# crictl ps 

解决步骤 #

1. 排查容器退出原因 #

定位停止容器的具体错误信息，执行以下命令获取日志：

# 查看停止容器的历史日志，替换<CONTAINER_ID>为目标容器ID
crictl logs <CONTAINER_ID>
# 或者通过kubectl查看对应Pod的上一次运行日志
kubectl logs kube-proxy-8t2mz -n kube-system --previous

重点关注日志中的内存不足、连接超时、配置错误等关键词。

2. 检查节点资源状态 #

容器随机停止最常见的原因是节点资源耗尽，执行以下命令查看CPU、内存、磁盘使用情况：

top
free -h
df -h

如果资源使用率接近阈值，需要：

• 清理节点上的无用进程或镜像
• 扩容节点硬件资源
• 调整Pod的资源请求/限制（比如修改kube-proxy的DaemonSet配置，增加resources.requests和resources.limits参数）

3. 统一Kubernetes组件镜像版本 #

当前集群中kube-controller-manager和kube-proxy存在v1.25.1和v1.25.2两个版本，版本不一致可能引发兼容性问题，统一使用同一版本：

1. 删除多余版本镜像：

crictl rmi registry.k8s.io/kube-controller-manager:v1.25.2 registry.k8s.io/kube-proxy:v1.25.2

2. 修改组件配置指定镜像版本：

# 修改kube-proxy的DaemonSet配置
kubectl edit daemonset kube-proxy -n kube-system
# 修改kube-controller-manager的Deployment配置
kubectl edit deployment kube-controller-manager -n kube-system

将spec.template.spec.containers[0].image统一设置为对应v1.25.1版本的镜像地址。

4. 检查容器运行时稳定性 #

确认containerd/docker等运行时是否正常，查看运行时日志：

# 使用containerd时
journalctl -u containerd -f
# 使用docker时
journalctl -u docker -f

如果运行时存在崩溃、连接异常等问题，重启运行时服务或升级到兼容Kubernetes 1.25的版本。

5. 检查kubelet状态与配置 #

kubelet负责节点Pod生命周期管理，查看其日志和配置：

# 实时查看kubelet日志
journalctl -u kubelet -f
# 检查kubelet服务状态
systemctl status kubelet

如果是kubelet驱逐策略导致容器停止，修改/var/lib/kubelet/config.yaml中的evictionHard参数，调高资源阈值。

6. 验证etcd健康状态 #

etcd不稳定会导致控制平面组件异常，执行以下命令检查etcd集群健康：

kubectl exec -it etcd-debian -n kube-system -- etcdctl endpoint health --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key

若etcd存在节点断开、数据不一致问题，需重新同步数据或替换故障节点。

内容的提问来源于stack exchange，提问作者PersianGulf