kubeadm加入Kubernetes集群超时，kubelet报文件系统缓存及PLEG异常问题求助

问题描述 #

我尝试将新节点加入Kubernetes集群，执行的命令是：

kubeadm join 172.29.217.209:6443 --token jew814.e5iofd6qdz26d9q --discovery-token-ca-cert-hash sha256:21919c93d652f17c6207ddfc86660bdf2f652d34b3c2a7fca6b056bcaca

但执行后出现超时错误，具体信息如下：

kubeadm join执行错误输出 #

[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
timed out waiting for the condition

This error is likely caused by:

• The kubelet is not running
• The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:

• 'systemctl status kubelet'
• 'journalctl -xeu kubelet'

error execution phase kubelet-start: timed out waiting for the condition
To see the stack trace of this error execute with --v=5 or higher

kubelet v1.28.3日志 #

Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.859742    2034 server.go:467] "Kubelet version" kubeletVersion="v1.28.3"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.859809    2034 server.go:469] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.860077    2034 server.go:630] "Standalone mode, no API client"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.876929    2034 server.go:518] "No api server defined - no events will be sent to API server"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.876952    2034 server.go:725] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.877119    2034 container_manager_linux.go:265] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.877260    2034 container_manager_linux.go:270] "Creating Container Manager object based on Node Config" nodeConfig={"RuntimeCgroupsName":"","SystemCgroupsName":"","KubeletCgroupsName":">
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.877280    2034 topology_manager.go:138] "Creating topology manager with none policy"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.877289    2034 container_manager_linux.go:301] "Creating device plugin manager"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.877377    2034 state_mem.go:36] "Initialized new in-memory state store"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.877438    2034 kubelet.go:399] "Kubelet is running in standalone mode, will skip API server sync"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.877860    2034 kuberuntime_manager.go:257] "Container runtime initialized" containerRuntime="containerd" version="1.6.20~ds1" apiVersion="v1"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.878030    2034 volume_host.go:74] "KubeClient is nil. Skip initialization of CSIDriverLister"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: W1107 22:50:19.878183    2034 csi_plugin.go:189] kubernetes.io/csi: kubeclient not set, assuming standalone kubelet
Nov 07 22:50:19 k8sslave01 kubelet[2034]: W1107 22:50:19.878201    2034 csi_plugin.go:266] Skipping CSINode initialization, kubelet running in standalone mode
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.878420    2034 server.go:1232] "Started kubelet"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.878556    2034 kubelet.go:1579] "No API server defined - no node status update will be sent"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.878828    2034 server.go:194] "Starting to listen read-only" address="0.0.0.0" port=10255
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.879608    2034 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.879895    2034 server.go:162] "Starting to listen" address="0.0.0.0" port=10250
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.880142    2034 ratelimit.go:65] "Setting rate limiting for podresources endpoint" qps=100 burstTokens=10
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.880608    2034 server.go:462] "Adding debug handlers to kubelet server"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.881041    2034 server.go:233] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: E1107 22:50:19.881520    2034 cri_stats_provider.go:448] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/containerd/io.contain>
Nov 07 22:50:19 k8sslave01 kubelet[2034]: E1107 22:50:19.881621    2034 kubelet.go:1431] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.881707    2034 volume_manager.go:291] "Starting Kubelet Volume Manager"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.883565    2034 desired_state_of_world_populator.go:151] "Desired state populator starts to run"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.883746    2034 reconciler_new.go:29] "Reconciler: start to sync state"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.901138    2034 kubelet_network_linux.go:50] "Initialized iptables rules." protocol="IPv4"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.905302    2034 kubelet_network_linux.go:50] "Initialized iptables rules." protocol="IPv6"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.905530    2034 status_manager.go:213] "Kubernetes client is nil, not starting status manager"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.905614    2034 kubelet.go:2303] "Starting kubelet main sync loop"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: E1107 22:50:19.905713    2034 kubelet.go:2327] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.906704    2034 cpu_manager.go:214] "Starting CPU manager" policy="none"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.906721    2034 cpu_manager.go:215] "Reconciling" reconcilePeriod="10s"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.906739    2034 state_mem.go:36] "Initialized new in-memory state store"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.908908    2034 policy_none.go:49] "None policy: Start"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.909341    2034 memory_manager.go:169] "Starting memorymanager" policy="None"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.909360    2034 state_mem.go:35] "Initializing new in-memory state store"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.911409    2034 manager.go:471] "Failed to read data from checkpoint" checkpoint="kubelet_internal_checkpoint" err="checkpoint is not found"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.911541    2034 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Nov 07 22:50:19 k8sslave01 kubelet[2034]: I1107 22:50:19.984679    2034 desired_state_of_world_populator.go:159] "Finished populating initial desired state of world"

集群主节点信息 #

[root@k8smasterone ~]# kubectl get nodes
NAME           STATUS   ROLES           AGE     VERSION
k8smasterone   Ready    control-plane   2y96d   v1.28.1

从节点kubelet状态 #

root@k8sslave01:~# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; preset: enabled)
Active: active (running) since Tue 2023-11-07 22:50:19 CST; 12min ago
Docs: https://kubernetes.io/docs/
Main PID: 2034 (kubelet)
Tasks: 10 (limit: 2025)
Memory: 38.0M
CPU: 3.959s
CGroup: /system.slice/kubelet