Kube-system组件反复CrashLoopBackOff问题求助

环境信息 #

• Ubuntu 22.04虚拟机
• Kubernetes版本v1.27.1
• Docker作为容器运行时
• 未安装CNI插件
• 磁盘、内存资源充足

当前状态 #

执行kubectl get pods --all-namespaces结果：

root@master:~# kubectl get pods --all-namespaces
NAMESPACE     NAME                                         READY   STATUS             RESTARTS        AGE
kube-system   coredns-5d78c9869d-t44fz                     0/1     Pending            0               12m
kube-system   coredns-5d78c9869d-vzbq2                     0/1     Pending            0               12m
kube-system   etcd-master.example.com                      1/1     Running            4 (5m53s ago)   12m
kube-system   kube-apiserver-master.example.com            1/1     Running            6 (5m10s ago)   13m
kube-system   kube-controller-manager-master.example.com   0/1     CrashLoopBackOff   6 (100s ago)    11m
kube-system   kube-proxy-ltwtp                             0/1     CrashLoopBackOff   6 (104s ago)    12m
kube-system   kube-scheduler-master.example.com            0/1     CrashLoopBackOff   6 (92s ago)     12m

kube-proxy日志：

root@master:~# kubectl logs kube-proxy-ltwtp -n kube-system
I0417 15:20:58.122219       1 node.go:141] Successfully retrieved node IP: 10.0.2.15
I0417 15:20:58.122427       1 server_others.go:110] "Detected node IP" address="10.0.2.15"
I0417 15:20:58.122498       1 server_others.go:551] "Using iptables proxy"
I0417 15:20:58.144359       1 server_others.go:190] "Using iptables Proxier"
I0417 15:20:58.144423       1 server_others.go:197] "kube-proxy running in dual-stack mode" ipFamily=IPv4
I0417 15:20:58.144430       1 server_others.go:198] "Creating dualStackProxier for iptables"
I0417 15:20:58.144435       1 server_others.go:465] "Detect-local-mode set to ClusterCIDR, but no cluster CIDR defined"
I0417 15:20:58.144438       1 server_others.go:521] "Defaulting to no-op detect-local" detectLocalMode="ClusterCIDR"
I0417 15:20:58.144467       1 proxier.go:253] "Setting route_localnet=1 to allow node-ports on localhost; to change this either disable iptables.localhostNodePorts (--iptables-localhost-nodeports) or set nodePortAddresses (--nodeport-addresses) to filter loopback addresses"
I0417 15:20:58.145393       1 server.go:657] "Version info" version="v1.27.1"
I0417 15:20:58.145691       1 server.go:659] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0417 15:20:58.148938       1 conntrack.go:52] "Setting nf_conntrack_max" nfConntrackMax=131072
I0417 15:20:58.151422       1 config.go:188] "Starting service config controller"
I0417 15:20:58.152780       1 shared_informer.go:311] Waiting for caches to sync for service config
I0417 15:20:58.152154       1 config.go:97] "Starting endpoint slice config controller"
I0417 15:20:58.152838       1 shared_informer.go:311] Waiting for caches to sync for endpoint slice config
I0417 15:20:58.152581       1 config.go:315] "Starting node config controller"
I0417 15:20:58.152843       1 shared_informer.go:311] Waiting for caches to sync for node config
I0417 15:20:58.253957       1 shared_informer.go:318] Caches are synced for node config
I0417 15:20:58.254322       1 shared_informer.go:318] Caches are synced for service config
I0417 15:20:58.254380       1 shared_informer.go:318] Caches are synced for endpoint slice config

后续问题：
一段时间后kubectl命令无法执行，报错：

root@master:~# kubectl get pods --all-namespaces
The connection to the server 10.0.2.15:6443 was refused - did you specify the right host or port?

root@master:~# kubectl get nodes
The connection to the server 10.0.2.15:6443 was refused - did you specify the right host or port?

重启虚拟机或等待后命令可恢复，但组件崩溃问题重复出现。

解决方案 #

1. 补全CNI插件安装 #

CoreDNS处于Pending状态是因为节点未就绪，先安装CNI插件：

• 以Calico为例，执行：

  kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml
  

  若无法访问外网，可下载yaml文件离线执行。

2. 排查控制器组件日志 #

kube-proxy日志无异常，需查看kube-controller-manager和kube-scheduler的崩溃日志：

# 查看当前日志
kubectl logs kube-controller-manager-master.example.com -n kube-system
kubectl logs kube-scheduler-master.example.com -n kube-system

# 查看历史崩溃日志
kubectl logs --previous kube-controller-manager-master.example.com -n kube-system
kubectl logs --previous kube-scheduler-master.example.com -n kube-system

3. 修复ClusterCIDR配置 #

kube-proxy日志提示Detect-local-mode set to ClusterCIDR, but no cluster CIDR defined，需补全集群网段配置：

• 编辑kube-controller-manager的静态pod文件（路径：/etc/kubernetes/manifests/kube-controller-manager.yaml），添加启动参数：

  - --cluster-cidr=192.168.0.0/16  # 需与CNI插件网段一致，Calico默认是192.168.0.0/16
  

• 保存后重启kubelet：

  systemctl restart kubelet
  

4. 检查API Server与etcd稳定性 #

etcd和API Server有多次重启记录，需验证etcd健康状态：

etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint health

同时查看API Server的pod定义，确认--etcd-servers参数配置正确：

kubectl describe pod kube-apiserver-master.example.com -n kube-system

5. 验证节点网络与主机名映射 #

确保虚拟机/etc/hosts中包含节点IP与主机名的映射：

10.0.2.15 master.example.com master

总结 #

核心问题大概率是ClusterCIDR未配置导致控制器组件异常，叠加API Server因etcd连接或网络问题反复重启，最终引发kubectl连接中断。优先补全CNI插件和ClusterCIDR配置，再根据控制器日志排查具体崩溃原因，即可解决重复崩溃问题。

内容的提问来源于stack exchange，提问作者J. Kim