Kuberhealthy部署健康检查频繁失败，报ClusterUnhealthy错误

[FIRING:2] kuberhealthy (ClusterUnhealthy kuberhealthy http kuberhealthy observability/kube-prometheus-stack-prometh

问题重现步骤 #

• Kuberhealthy定期执行部署检查，部署看似完成，但无法向Kuberhealthy服务上报状态

集群事件日志 #

$ k get events -nkuberhealthy | grep deployment | tail
12m         Normal    ScalingReplicaSet        deployment/deployment-deployment                      将副本集deployment-deployment-XXX缩容至2个副本
12m         Normal    ScalingReplicaSet        deployment/deployment-deployment                      将副本集deployment-deployment-XXX扩容至4个副本
12m         Normal    ScalingReplicaSet        deployment/deployment-deployment                      将副本集deployment-deployment-XXX缩容至0个副本
3m31s       Normal    ScalingReplicaSet        deployment/deployment-deployment                      将副本集deployment-deployment-XXX扩容至4个副本
3m9s        Normal    ScalingReplicaSet        deployment/deployment-deployment                      将副本集deployment-deployment-XXX扩容至2个副本
3m9s        Normal    ScalingReplicaSet        deployment/deployment-deployment                      将副本集deployment-deployment-69459d778b缩容至2个副本
3m9s        Normal    ScalingReplicaSet        deployment/deployment-deployment                      将副本集deployment-deployment-XXX扩容至4个副本
3m          Normal    ScalingReplicaSet        deployment/deployment-deployment                      将副本集deployment-deployment-XXX缩容至0个副本
63m         Warning   FailedToUpdateEndpoint   endpoints/deployment-svc                              更新端点kuberhealthy/deployment-svc失败：无法完成端点"deployment-svc"的操作：对象已被修改；请基于最新版本应用更改后重试
53m         Warning   FailedToUpdateEndpoint

调试日志 #

$ k logs deployment-XXX -nkuberhealthy
time="2022-12-16T12:36:43Z" level=info msg="发现实例命名空间：kuberhealthy"
time="2022-12-16T12:36:43Z" level=info msg="Kuberhealthy位于kuberhealthy命名空间中。"
time="2022-12-16T12:36:43Z" level=info msg="已启用调试日志。"
time="2022-12-16T12:36:43Z" level=debug msg="[/app/deployment-check]"
time="2022-12-16T12:36:43Z" level=info msg="解析CHECK_IMAGE：XXXX"
time="2022-12-16T12:36:43Z" level=info msg="解析CHECK_IMAGE_ROLL_TO：XXX"
time="2022-12-16T12:36:43Z" level=info msg="发现Pod命名空间：kuberhealthy"
time="2022-12-16T12:36:43Z" level=info msg="将在kuberhealthy命名空间中执行检查。"
time="2022-12-16T12:36:43Z" level=info msg="解析CHECK_DEPLOYMENT_REPLICAS：2"
time="2022-12-16T12:36:43Z" level=info msg="解析CHECK_SERVICE_ACCOUNT：default"
time="2022-12-16T12:36:43Z" level=info msg="检查时间限制设置为：14m46.760673918s"
time="2022-12-16T12:36:43Z" level=info msg="解析CHECK_DEPLOYMENT_ROLLING_UPDATE：true"
time="2022-12-16T12:36:43Z" level=info msg="检查部署镜像将从[XXX]滚动更新至[XXXX]"
time="2022-12-16T12:36:43Z" level=debug msg="允许此次检查在14m46.760673918s内完成。"
time="2022-12-16T12:36:43Z" level=info msg="已创建Kubernetes客户端。"
time="2022-12-16T12:36:43Z" level=info msg="等待节点就绪后开始检查。"
time="2022-12-16T12:36:43Z" level=debug msg="检查kuberhealthy端点：XXX是否就绪。"
time="2022-12-16T12:36:43Z" level=debug msg="XXX。"
time="2022-12-16T12:36:43Z" level=debug msg="Kuberhealthy端点：XXX已就绪。继续执行检查。"
time="2022-12-16T12:36:43Z" level=info msg="开始检查。"
time="2022-12-16T12:36:43Z" level=info msg="清理所有属于此次检查的孤立资源。"
time="2022-12-16T12:36:43Z" level=info msg="尝试查找此前创建的属于此次检查的服务。"
time="2022-12-16T12:36:43Z" level=debug msg="找到1个服务。"
time="2022-12-16T12:36:43Z" level=debug msg="服务：kuberhealthy"
time="2022-12-16T12:36:43Z" level=info msg="未找到属于此次检查的旧服务。"
time="2022-12-16T12:36:43Z" level=info msg="尝试查找此前创建的属于此次检查的部署。"
time="2022-12-16T12:36:44Z" level=debug msg="找到1个部署"
time="2022-12-16T12:36:44Z" level=debug msg=kuberhealthy
time="2022-12-16T12:36:44Z" level=info msg="未找到属于此次检查的旧部署。"
time="2022-12-16T12:36:44Z" level=info msg="已成功清理之前的检查资源。"
time="2022-12-16T12:36:44Z" level=info msg="在kuberhealthy命名空间中创建包含2个副本的部署资源，使用镜像XXX]"
time="2022-12-16T12:36:44Z" level=info msg="使用镜像[XXX]创建容器"
time="2022-12-16T12:36:44Z" level=info msg="已创建部署资源。"
time="2022-12-16T12:36:44Z" level=info msg="在集群中创建名为deployment-deployment的部署。"
time="2022-12-16T12:36:44Z" level=info msg="等待部署创建完成。"
time="2022-12-16T12:36:44Z" level=debug msg="收到部署变更事件：deployment-deployment触发ADDED事件"
time="2022-12-16T12:36:47Z" level=debug msg="收到部署变更事件：deployment-deployment触发MODIFIED事件"
time="2022-12-16T12:36:48Z" level=debug msg="收到部署变更事件：deployment-deployment触发MODIFIED事件"
time="2022-12-16T12:36:53Z" level=debug msg="收到部署变更事件：deployment-deployment触发MODIFIED事件"
time="2022-12-16T12:36:53Z" level=info msg="部署报告状态为Available：True。"
time="2022-12-16T12:36:53Z" level=info msg="已在kuberhealthy命名空间中创建部署：deployment-deployment"
time="2022-12-16T12:36:53Z" level=info msg="为kuberhealthy命名空间创建服务资源。"
time="2022-12-16T12:36:53Z" level=info msg="已创建服务资源。"
time="2022-12-16T12:36:53Z" level=info msg="在集群中创建名为deployment-svc的服务。"
time="2022-12-16T12:36:53Z" level=info msg="等待服务创建完成。"
time="2022-12-16T12:36:53Z" level=debug msg="收到服务变更事件：ADDED"
time="2022-12-16T12:36:53Z" level=info msg="找到集群IP：XXX"
time="2022-12-16T12:36:53Z" level=info msg="已在kuberhealthy命名空间中创建服务：deployment-svc"
time="2022-12-16T12:36:53Z" level=debug msg="获取属于deployment-svc的集群IP"
time="2022-12-16T12:36:53Z" level=info msg="找到服务集群IP地址：XXX"
time="2022-12-16T12:36:53Z" level=info msg="等待端点响应。"
time="2022-12-16T12:36:53Z" level=debug msg="设置退避循环超时时间：3m0s"
time="2022-12-16T12:36:53Z" level=info msg="开始HTTP GET请求的退避循环。"
time="2022-12-16T12:36:53Z" level=debug msg="向XXX发送GET请求"
time="2022-12-16T12:36:53Z" level=debug msg="收到401响应"
time="2022-12-16T12:36:53Z" level=info msg="5秒后重试。"
time="2022-12-16T12:36:58Z" level=error msg="向集群内服务发送请求时出错：无法从指定地址获取响应：XXX"
time="2022-12-16T12:36:58Z" level=info msg="清理部署和服务资源。"
time="2022-12-16T12:36:58Z" level=info msg="尝试删除kuberhealthy命名空间中的deployment-svc服务。"
time="2022-12-16T12:36:58Z" level=debug msg="检查服务是否已删除。"
time="2022-12-16T12:36:58Z" level=debug msg="删除服务的等待操作尚未超时。"
time="2022-12-16T12:36:58Z" level=debug msg="等待5秒后重试。"
time="2022-12-16T12:37:03Z" level=info msg="尝试删除kuberhealthy命名空间中的部署。"
time="2022-12-16T12:37:03Z" level=debug msg="检查部署是否已删除。"
time="2022-12-16T12:37:03Z" level=debug msg="删除部署的等待操作尚未超时。"
time="2022-12-16T12:37:03Z" level=debug msg="等待5秒后重试。"
time="2022-12-16T12:37:08Z" level=info msg="清理流程完成。"
time="2022-12-16T12:37:08Z" level=error msg="向Kuberhealthy上报错误：[无法从指定地址获取响应：XXX"

内容的提问来源于stack exchange，提问作者Daemon