Knative间歇性无法创建Deployment问题排查求助

Knative会间歇性无法创建新的Deployment，数小时后又会自动恢复并完成创建。在此期间，Serving组件持续出现报错，推测是对Kubernetes服务的请求超时，但无法确定具体原因。

预期行为 #

• 更新服务时，新Revision的部署能够正常完成

实际行为 #

• 偶尔进行有效变更（例如修改注解值）时，Knative无法部署新Revision，陷入持续reconcile的状态，数小时后才自动恢复
• 执行kn revision list -A输出如下：

$ kn revision list -A
NAMESPACE        NAME                       SERVICE              TRAFFIC   TAGS      GENERATION   AGE         CONDITIONS   READY     REASON
knative       service-00033                  service                                 33           <invalid>   0 OK / 3     Unknown   Deploying
knative       service-00032                  service             100%      primary   32           <invalid>   4 OK / 4     True

相关日志 #

Controller超时错误日志 #

{
  "insertId": "plhs429mzmf9nh5f",
  "jsonPayload": {
    "logger": "controller.event-broadcaster",
    "caller": "record/event.go:285",
    "knative.dev/pod": "controller-8c6b99cb7-7zg6n",
    "commit": "484e848",
    "message": "Event(v1.ObjectReference{Kind:\"Revision\", Namespace:\"knative\", Name:\"service-00033\", UID:\"8a09a3ff-655e-4e5f-b8d4-1a4886ab0678\", APIVersion:\"serving.knative.dev/v1\", ResourceVersion:\"1844291799\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' failed to create deployment \"service-api-00033-deployment\": Post \"https://10.123.20.1:443/apis/apps/v1/namespaces/knative/deployments\": context deadline exceeded",
    "timestamp": "2023-06-30T09:57:08.7332053Z"
  }
}

Webhook验证错误日志 #

{
  "insertId": "k078pd2dmx16qrr7",
  "jsonPayload": {
    "knative.dev/pod": "webhook-d44b476b8-89gbx",
    "message": "Failed the resource specific validation",
    "knative.dev/operation": "UPDATE",
    "logger": "webhook",
    "knative.dev/name": "service",
    "knative.dev/subresource": "",
    "knative.dev/namespace": "knative",
    "knative.dev/kind": "serving.knative.dev/v1, Kind=Service",
    "knative.dev/resource": "serving.knative.dev/v1, Resource=services",
    "commit": "484e848",
    "knative.dev/userinfo": "system:serviceaccount:service:default",
    "timestamp": "2023-06-30T09:56:38.327880939Z",
    "caller": "validation/validation_admit.go:183",
    "stacktrace": "knative.dev/pkg/webhook/resourcesemantics/validation.validate\n\tknative.dev/pkg@v0.0.0-20230117181655-247510c00e9d/webhook/resourcesemantics/validation/validation_admit.go:183\nknative.dev/pkg/webhook/resourcesemantics/validation.(*reconciler).Admit\n\tknative.dev/pkg@v0.0.0-20230117181655-247510c00e9d/webhook/resourcesemantics/validation/validation_admit.go:79\nknative.dev/pkg/webhook.admissionHandler.func1\n\tknative.dev/pkg@v0.0.0-20230117181655-247510c00e9d/webhook/admission.go:123\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2109\nnet/http.(*ServeMux).ServeHTTP\n\tnet/http/server.go:2487\nknative.dev/pkg/webhook.(*Webhook).ServeHTTP\n\tknative.dev/pkg@v0.0.0-20230117181655-247510c00e9d/webhook/webhook.go:263\nknative.dev/pkg/network/handlers.(*Drainer).ServeHTTP\n\tknative.dev/pkg@v0.0.0-20230117181655-247510c00e9d/network/handlers/drain.go:113\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2947\nnet/http.(*conn).serve\n\tnet/http/server.go:1991"
  }
}

排查方向 #

• 检查Kubernetes API Server状态：查看故障时段API Server的CPU、内存使用率，是否存在请求堆积、响应超时，确认是否有性能瓶颈
• 验证网络连通性：在Knative Controller Pod内测试与API Server IP（https://10.123.20.1:443）的连通性，检查是否有间歇性丢包、延迟过高的情况
• 细化Webhook日志：调整Webhook日志级别，获取Failed the resource specific validation的具体失败原因，确认是否与校验时的API请求超时相关
• 核对版本兼容性：确认Knative Serving与Kubernetes版本是否匹配，排查是否存在已知的兼容性bug
• 检查集群资源：故障时段查看节点的CPU、内存、磁盘使用率，是否有节点资源耗尽导致API Server响应缓慢
• 排查ETCD状态：ETCD性能会直接影响API Server，检查ETCD的读写延迟、磁盘IO是否正常

内容的提问来源于stack exchange，提问作者Desolar1um