Rancher集群创建停滞：等待Cluster Agent连接失败

通过Docker（Podman）在RedHat 8.9上部署Rancher 2.8.3集群，使用默认配置创建新集群后，执行注册命令添加节点，节点在控制台可见但一直卡在等待cluster agent注册状态。待注册节点的rancher-system-agent日志持续提示secret过期。

集群配置日志 #

[INFO ] waiting for at least one control plane, etcd, and worker node to be registered
[INFO ] configuring bootstrap node(s) custom-8b637c9cd841: waiting for bootstrap etcd to be available
[INFO ] configuring bootstrap node(s) custom-8b637c9cd841: waiting for agent to check in and apply initial plan
[INFO ] configuring bootstrap node(s) custom-8b637c9cd841: waiting for probes: calico, etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet
[INFO ] configuring bootstrap node(s) custom-8b637c9cd841: waiting for probes: calico, kube-apiserver, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) custom-8b637c9cd841: waiting for probes: calico
[INFO ] configuring bootstrap node(s) custom-8b637c9cd841: waiting for cluster agent to connect

Rancher-System-Agent日志 #

rancher-system-agent[13135]: time="2024-04-30T14:27:40+02:00" level=info msg="[Applyinator] Running command: sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null]"
rancher-system-agent[13135]: time="2024-04-30T14:27:40+02:00" level=info msg="[12cb26ca54456920e77a29191a815c3d6fed7af28c7a8c072a4d5c214607e6fe_0:stdout]: Name Location Size Created"
rancher-system-agent[13135]: time="2024-04-30T14:27:40+02:00" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"
rancher-system-agent[13135]: time="2024-04-30T14:27:40+02:00" level=info msg="[K8s] updated plan secret fleet-default/custom-8b637c9cd841-machine-plan with feedback"
rancher-system-agent[13135]: time="2024-04-30T14:27:45+02:00" level=error msg="[K8s] received secret to process that was older than the last secret operated on. (14820 vs 18024)"
rancher-system-agent[13135]: time="2024-04-30T14:27:45+02:00" level=error msg="error syncing 'fleet-default/custom-8b637c9cd841-machine-plan': handler secret-watch: secret received was too old, requeuing"
rancher-system-agent[13135]: W0430 14:29:05.831181   13135 reflector.go:456] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 35; INTERNAL_ERROR; received from peer") has prevented the request from succeeding

疑问 #

该问题是否由secret异常导致？

是，该问题直接由secret异常导致，具体细节和解决方式如下：

1. 异常根源

   • 日志中明确出现received secret to process that was older than the last secret operated on错误，说明节点上的rancher-system-agent收到了比已处理版本更旧的secret，导致无法正常同步集群配置。
   • 后续的watch of *v1.Secret ended with: an error on the server错误，说明节点与Rancher服务器之间的secret监听流出现通信故障，进一步阻碍了同步流程。

2. 解决步骤

   • 清理节点旧残留：在待注册节点上停止rancher-system-agent服务，清理/var/lib/rancher/system-agent/路径下的缓存文件，再重启服务。
   • 重新生成注册命令：在Rancher控制台删除当前停滞的节点，重新生成新的注册命令并在节点上执行，确保使用最新的secret信息。
   • 检查时间同步：确认所有节点与Rancher服务器的系统时间一致，时间偏差会触发secret过期误判。
   • 验证网络连通性：确保节点与Rancher服务器之间网络稳定，无防火墙或代理拦截Kubernetes API的watch请求。

内容的提问来源于stack exchange，提问作者Okeano