重建GKE集群后，单个LoadBalancer服务无法访问的原因排查

我在GKE集群中部署了两个LoadBalancer类型的服务，其中一个可通过外部负载均衡IP正常访问，另一个则无法访问。集群删除重建前两个服务均正常可用，新集群通过Terraform创建，配置与旧集群一致，且使用同一Kustomize目录的K8s配置重新部署了资源。

正常工作的服务 #

Name:                     v2alpha-public-api-server
Namespace:                default
Labels:                   app.kubernetes.io/component=kingdom
                          app.kubernetes.io/name=v2alpha-public-api-server
                          app.kubernetes.io/part-of=halo-cmms
Annotations:              cloud.google.com/app-protocols: {"grpc-port": "HTTP2"}
                          cloud.google.com/neg: {"ingress":true}
                          kubernetes.io/ingress.allow-http: false
                          system: kingdom
Selector:                 app=v2alpha-public-api-server-app
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.92.11.149
IPs:                      10.92.11.149
IP:                       34.123.211.251
LoadBalancer Ingress:     34.123.211.251
Port:                     grpc-port  8443/TCP
TargetPort:               8443/TCP
NodePort:                 grpc-port  31562/TCP
Endpoints:                10.88.1.3:8443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

无法访问的服务 #

Name:                     system-api-server
Namespace:                default
Labels:                   app.kubernetes.io/component=kingdom
                          app.kubernetes.io/name=system-api-server
                          app.kubernetes.io/part-of=halo-cmms
Annotations:              cloud.google.com/app-protocols: {"grpc-port": "HTTP2"}
                          cloud.google.com/neg: {"ingress":true}
                          kubernetes.io/ingress.allow-http: false
                          system: kingdom
Selector:                 app=system-api-server-app
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.92.1.31
IPs:                      10.92.1.31
IP:                       34.69.224.217
LoadBalancer Ingress:     34.69.224.217
Port:                     grpc-port  8443/TCP
TargetPort:               8443/TCP
NodePort:                 grpc-port  32064/TCP
Endpoints:                10.88.1.2:8443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

已排查情况 #

• 集群为VPC-native、公共、单可用区类型
• 通过端口转发可访问故障服务，说明底层Deployment运行正常
• 删除集群网络策略后问题仍存在，且网络策略日志显示连接已被允许
• 防火墙日志确认连接通过了防火墙规则
• 使用grpc_health_probe测试：故障服务提示timeout: failed to connect service，延长超时时间无效；正常服务可正常收到响应

内容的提问来源于stack exchange，提问作者Sanjay Vasandani