同一份Kong部署YAML在GKE正常,MacOS本地MicroK8s探针失败
Kong在MacOS MicroK8s环境中探针超时问题排查
问题背景
相同的Kong Deployment YAML在GKE集群部署后运行正常,但在MacOS本地MicroK8s环境中,Pod状态显示Running,但就绪探针(Readiness Probe)和存活探针(Liveness Probe)频繁出现超时失败警告,报错信息为:
Readiness probe failed: Get "http://10.1.254.79:8100/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
部署YAML
apiVersion: apps/v1 kind: Deployment metadata: name: local-test-kong labels: app: local-test-kong spec: replicas: 1 selector: matchLabels: app: local-test-kong strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 0 type: RollingUpdate template: metadata: labels: app: local-test-kong spec: automountServiceAccountToken: false containers: - envFrom: - configMapRef: name: kong-env-vars image: kong:2.6 imagePullPolicy: IfNotPresent lifecycle: preStop: exec: command: - /bin/sh - -c - /bin/sleep 15 && kong quit livenessProbe: failureThreshold: 3 httpGet: path: /status port: status scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 name: proxy ports: - containerPort: 8000 name: proxy protocol: TCP - containerPort: 8100 name: status protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /status port: status scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 resources: # ToDo limits: cpu: 256m memory: 256Mi requests: cpu: 256m memory: 256Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /kong_prefix/ name: kong-prefix-dir - mountPath: /tmp name: tmp-dir - mountPath: /kong_dbless/ name: kong-custom-dbless-config-volume terminationGracePeriodSeconds: 30 volumes: - name: kong-prefix-dir - name: tmp-dir - configMap: defaultMode: 0555 name: kong-declarative name: kong-custom-dbless-config-volume
GKE环境Pod状态(无异常)
➜ kubectl get pods NAME READY STATUS RESTARTS AGE local-test-kong-678598ffc6-ll9s8 1/1 Running 0 25m ➜ kubectl describe pod/local-test-kong-678598ffc6-ll9s8 Name: local-test-kong-678598ffc6-ll9s8 Namespace: local-test-kong Priority: 0 Node: gke-paas-cluster-prd-tf9-default-pool-e7cb502a-ggxl/10.128.64.95 Start Time: Wed, 23 Nov 2022 00:12:56 +0800 Labels: app=local-test-kong pod-template-hash=678598ffc6 Annotations: kubectl.kubernetes.io/restartedAt: 2022-11-23T00:12:56+08:00 Status: Running IP: 10.128.96.104 IPs: IP: 10.128.96.104 Controlled By: ReplicaSet/local-test-kong-678598ffc6 Containers: proxy: Container ID: containerd://1bd392488cfe33dcc62f717b3b8831349e8cf573326add846c9c843c7bf15e2a Image: kong:2.6 Image ID: docker.io/library/kong@sha256:62eb6d17133b007cbf5831b39197c669b8700c55283270395b876d1ecfd69a70 Ports: 8000/TCP, 8100/TCP Host Ports: 0/TCP, 0/TCP State: Running Started: Wed, 23 Nov 2022 00:12:58 +0800 Ready: True Restart Count: 0 Limits: cpu: 256m memory: 256Mi Requests: cpu: 256m memory: 256Mi Liveness: http-get http://:status/status delay=10s timeout=5s period=10s #success=1 #failure=3 Readiness: http-get http://:status/status delay=10s timeout=5s period=10s #success=1 #failure=3 Environment Variables from: kong-env-vars ConfigMap Optional: false Environment: <none> Mounts: /kong_dbless/ from kong-custom-dbless-config-volume (rw) /kong_prefix/ from kong-prefix-dir (rw) /tmp from tmp-dir (rw) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: kong-prefix-dir: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> tmp-dir: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> kong-custom-dbless-config-volume: Type: ConfigMap (a volume populated by a ConfigMap) Name: kong-declarative Optional: false QoS Class: Guaranteed Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 25m default-scheduler Successfully assigned local-test-kong/local-test-kong-678598ffc6-ll9s8 to gke-paas-cluster-prd-tf9-default-pool-e7cb502a-ggxl Normal Pulled 25m kubelet Container image "kong:2.6" already present on machine Normal Created 25m kubelet Created container proxy Normal Started 25m kubelet Started container proxy
MacOS MicroK8s环境Pod状态(探针失败)
➜ kubectl get pods NAME READY STATUS RESTARTS AGE local-test-kong-54cfc585cb-7grj8 1/1 Running 0 86s ➜ kubectl describe pod/local-test-kong-54cfc585cb-7grj8 Name: local-test-kong-54cfc585cb-7grj8 Namespace: local-test-kong Priority: 0 Node: microk8s-vm/192.168.64.5 Start Time: Wed, 23 Nov 2022 00:39:33 +0800 Labels: app=local-test-kong pod-template-hash=54cfc585cb Annotations: cni.projectcalico.org/podIP: 10.1.254.79/32 cni.projectcalico.org/podIPs: 10.1.254.79/32 kubectl.kubernetes.io/restartedAt: 2022-11-23T00:39:33+08:00 Status: Running IP: 10.1.254.79 IPs: IP: 10.1.254.79 Controlled By: ReplicaSet/local-test-kong-54cfc585cb Containers: proxy: Container ID: containerd://d60d09ca8b77ee59c80ea060dcb651c3e346c3a5f0147b0d061790c52193d93d Image: kong:2.6 Image ID: docker.io/library/kong@sha256:62eb6d17133b007cbf5831b39197c669b8700c55283270395b876d1ecfd69a70 Ports: 8000/TCP, 8100/TCP Host Ports: 0/TCP, 0/TCP State: Running Started: Wed, 23 Nov 2022 00:39:37 +0800 Ready: True Restart Count: 0 Limits: cpu: 256m memory: 256Mi Requests: cpu: 256m memory: 256Mi Liveness: http-get http://:status/status delay=10s timeout=5s period=10s #success=1 #failure=3 Readiness: http-get http://:status/status delay=10s timeout=5s period=10s #success=1 #failure=3 Environment Variables from: kong-env-vars ConfigMap Optional: false Environment: <none> Mounts: /kong_dbless/ from kong-custom-dbless-config-volume (rw) /kong_prefix/ from kong-prefix-dir (rw) /tmp from tmp-dir (rw) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: kong-prefix-dir: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> tmp-dir: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> kong-custom-dbless-config-volume: Type: ConfigMap (a volume populated by a ConfigMap) Name: kong-declarative Optional: false QoS Class: Guaranteed Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 92s default-scheduler Successfully assigned local-test-kong/local-test-kong-54cfc585cb-7grj8 to microk8s-vm Normal Pulled 90s kubelet Container image "kong:2.6" already present on machine Normal Created 90s kubelet Created container proxy Normal Started 89s kubelet Started container proxy Warning Unhealthy 68s kubelet Readiness probe failed: Get "http://10.1.254.79:8100/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Warning Unhealthy 68s kubelet Liveness probe failed: Get "http://10.1.254.79:8100/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
排查方向及解决方案
1. 调整探针启动延迟和超时时间
本地MicroK8s的VM资源通常比GKE节点有限,Kong加载配置和启动服务的时间更长,当前initialDelaySeconds:10可能不足以让状态端口8100完全就绪。
- 修改Deployment中的探针配置:
livenessProbe: initialDelaySeconds: 30 timeoutSeconds: 10 readinessProbe: initialDelaySeconds: 30 timeoutSeconds: 10 - 重启Deployment:
kubectl rollout restart deployment/local-test-kong -n local-test-kong
2. 检查Kong状态端口绑定配置
确认kong-env-vars ConfigMap中,Kong的状态监听地址是否绑定到0.0.0.0:8100,如果仅绑定到localhost,kubelet无法从外部访问该端口。
- 编辑ConfigMap:
kubectl edit configmap/kong-env-vars -n local-test-kong - 添加或修改变量:
data: KONG_STATUS_LISTEN: "0.0.0.0:8100" - 重启Deployment生效。
3. 排查MicroK8s网络插件限制
MicroK8s默认使用Calico作为CNI插件,可能存在网络策略阻止kubelet访问Pod端口。
- 临时删除命名空间下所有网络策略测试:
kubectl delete networkpolicy --all -n local-test-kong - 如果恢复正常,需要调整Calico网络策略,允许kubelet所在节点访问Pod的8100端口。
4. 优化EmptyDir卷性能
Deployment中kong-prefix-dir和tmp-dir使用默认EmptyDir(磁盘介质),本地VM磁盘性能可能较差,导致Kong启动延迟。
- 修改为内存介质的EmptyDir:
volumes: - name: kong-prefix-dir emptyDir: medium: Memory - name: tmp-dir emptyDir: medium: Memory - 重启Deployment生效。
5. 测试Pod内部端口连通性
进入Pod内部测试状态端口是否正常监听:
kubectl exec -it local-test-kong-54cfc585cb-7grj8 -n local-test-kong -- curl http://localhost:8100/status
如果返回正常状态,说明问题出在外部网络访问;如果也超时,说明Kong本身未正常启动状态服务,需要检查日志:
kubectl logs local-test-kong-54cfc585cb-7grj8 -n local-test-kong
内容的提问来源于stack exchange,提问作者Rakib
相关产品推荐
相关产品推荐

