Rancher部署的K3S集群cert-manager webhook调用超时问题求助

集群信息 #

• Kubernetes版本：v1.24.7+k3s1
• 部署方式：Rancher 2.7
• 主机操作系统：RHEL8

已尝试操作 #

• 使用kubectl重新部署cert-manager；
• 参考cert-manager官方验证指南进行测试，仍出现相同错误：

Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": context deadline exceeded

问题详情 #

通过Rancher部署K3S集群后，使用自定义镜像（来自Artifactory私有仓库）安装cert-manager v1.11.0，所有Pod均显示运行状态，但安装Helm Chart或部署资源时出现如下错误：

Error: Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": context deadline exceeded )

同时，cert-manager-webhook Pod日志显示无法访问Kubernetes API，出现I/O超时：

Trace[1068908304]: [30.003276269s] [30.003276269s] ENDE0314 15:02:02.236947       1 reflector.go:140] k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915": dial tcp 10.43.0.1:443: i/o timeout

W0314 15:03:28.953687       1 reflector.go:424] k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1.Secret: Get "https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915": dial tcp 10.43.0.1:443: i/o timeout

I0314 15:03:28.953816       1 trace.go:219] Trace[516939538]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169 (14-Mar-2023 15:02:58.949) (total time: 30004ms):Trace[516939538]: ---"Objects listed" error:Get "https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915": dial tcp 10.43.0.1:443: i/o timeout 30004ms

(15:03:28.953)Trace[516939538]: [30.004226263s] [30.004226263s] ENDE0314 15:03:28.953837       1 reflector.go:140] k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915": dial tcp 10.43.0.1:443: i/o timeout

W0314 15:04:44.919380       1 reflector.go:424] k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1.Secret: Get "https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915": dial tcp 10.43.0.1:443: i/o timeout

I0314 15:04:44.919458       1 trace.go:219] Trace[430405071]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169 (14-Mar-2023 15:04:14.918) (total time: 30000ms):Trace[430405071]: ---"Objects listed" error:Get "https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915": dial tcp 10.43.0.1:443: i/o timeout 30000ms (15:04:44.919)Trace[430405071]: [30.000964846s] [30.000964846s] ENDE0314 15:04:44.919472       1 reflector.go:140] k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915": dial tcp 10.43.0.1:443: i/o timeout

此外，通过Helm部署的actions-runner-controller Pod出现证书挂载失败：

Warning  FailedMount  17m                  kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[kube-api-access-v48zj secret tmp cert]: timed out waiting for the condition

Warning  FailedMount  8m32s                kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[tmp cert kube-api-access-v48zj secret]: timed out waiting for the condition

Warning  FailedMount  6m18s (x5 over 19m)  kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[secret tmp cert kube-api-access-v48zj]: timed out waiting for the condition

Warning  FailedMount  103s (x2 over 4m1s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[cert kube-api-access-v48zj secret tmp]: timed out waiting for the condition

Warning  FailedMount  86s (x18 over 21m)   kubelet            MountVolume.SetUp failed for volume "cert" : secret "actions-runner-controller-serving-cert" not found

关键提示：该问题在本地minikube环境中可正常运行，当前K3S集群处于隔离环境。

内容的提问来源于stack exchange，提问作者Saitama