GKE Ingress对接gRPC服务遇502错误：无法连接后端求助

问题描述 #

在GKE中配置Ingress对接gRPC服务时，使用grpcurl发起请求始终返回502 Bad Gateway错误，请求未到达Nginx实例。错误信息如下：

rpc error: code = Unavailable desc = failed to query for service descriptor "XXX": unexpected HTTP status code received from server: 502 (Bad Gateway)

从负载均衡日志中看到statusDetails: failed_to_connect_to_backend，确认错误由Ingress直接返回。

相关配置信息 #

服务与Deployment配置 #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: drm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: "nginx"
          imagePullPolicy: IfNotPresent
          resources:
            requests:
              cpu: "50m"
              memory: "100M"
            limits :
              cpu: "50m"
              memory: "300M"
          ports:
            - containerPort: 8080
            - containerPort: 9900
            - containerPort: 2222
          volumeMounts:
            - mountPath: /etc/nginx/templates
              name: nginx-conf
          livenessProbe:
            httpGet:
              path: /health
              port: 2222
          readinessProbe:
            httpGet:
              path: /health
              port: 2222
      volumes:
        - name: nginx-conf
          projected:
            sources:
            - configMap:
                name: nginx
                items:
                  - key: test
                    path: back.conf.template
            - configMap:
                name: nginx
                items:
                  - key: grpc
                    path: grpc.conf.template
            - configMap:
                name: nginx
                items:
                  - key: health
                    path: health.conf.template
---
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  namespace: drm
  name: http-hc-config
spec:
  healthCheck:
    checkIntervalSec: 60
    requestPath: /health
    type: HTTP
    port: 2222
---
apiVersion: v1
kind: Service
metadata:
  namespace: drm
  name: nginx
  annotations:
    cloud.google.com/backend-config: '{"default":"http-hc-config"}'
    cloud.google.com/app-protocols: '{"grpc":"HTTP2"}'
    cloud.google.com/neg: '{"ingress": true}'
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      name: back
    - protocol: TCP
      port: 2222
      targetPort: 2222
      name: health
  selector:
    app: nginx
  type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
  namespace: drm
  name: nginx-grpc
  annotations:
    cloud.google.com/backend-config: '{"default":"http-hc-config"}'
    cloud.google.com/app-protocols: '{"grpc":"HTTP2"}'
spec:
  ports:
    - protocol: TCP
      port: 9900
      targetPort: 9900
      name: grpc
    - protocol: TCP
      port: 2222
      targetPort: 2222
      name: health
  selector:
    app: nginx
  type: NodePort

Ingress配置 #

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
   name: ingress-resource-backend
   namespace: drm
   annotations:
     kubernetes.io/ingress.global-static-ip-name: piloto
     networking.gke.io/managed-certificates:  managed-cert-grpc, managed-cert-server
     kubernetes.io/ingress.class: "gce"
spec:
  rules:
    - host: tms-grpc.apps2go.tech
      http:
       paths:
       - path: "/*"
         pathType: ImplementationSpecific
         backend:
           service:
             name: nginx-grpc
             port:
               number: 9900
    - host: tms-server.apps2go.tech
      http:
        paths:
          - path: "/*"
            pathType: ImplementationSpecific
            backend:
              service:
                name: nginx
                port:
                  number: 8080

Nginx gRPC配置 #

server {
      listen 9900 http2;
      grpc_read_timeout 1800s;
      grpc_connect_timeout 1800s;
      grpc_send_timeout 1800s;
      location / {
          grpc_pass grpc://communication-terminal-grpc:9903;
      }
  }

负载均衡错误日志 #

{
  "insertId": "1vfbofdfbw4nl9",
  "jsonPayload": {
    "@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
    "cacheDecision": [
      "RESPONSE_HAS_CONTENT_TYPE",
      "CACHE_MODE_USE_ORIGIN_HEADERS"
    ],
    "remoteIp": "X.X.X.X",
    "statusDetails": "failed_to_connect_to_backend"
  },
  "httpRequest": {
    "requestMethod": "POST",
    "requestUrl": "XXXXX",
    "requestSize": "64",
    "status": 502,
    "responseSize": "392",
    "userAgent": "grpcurl/1.8.6 grpc-go/1.44.1-dev",
    "remoteIp": "x.X.X.X",
    "serverIp": "10.68.3.6",
    "latency": "9.038340s"
  },
  "resource": {
    "type": "http_load_balancer",
    "labels": {
      "forwarding_rule_name": "k8s2-fs-nz16btey-drm-ingress-resource-backend-9257uxp9",
      "project_id": "XXXXX",
      "url_map_name": "k8s2-um-nz16btey-drm-ingress-resource-backend-9257uxp9",
      "backend_service_name": "k8s1-ece260cf-drm-nginx-grpc-test-9900-0f35392e",
      "target_proxy_name": "k8s2-ts-nz16btey-drm-ingress-resource-backend-9257uxp9",
      "zone": "global"
    }
  },
  "timestamp": "2023-05-11T21:21:12.946016Z",
  "severity": "WARNING",
  "logName": "projects/XXXX/logs/requests",
  "trace": "projects/XXX/traces/15af45a6f3cfd121f8d7f5ccc043992d",
  "receiveTimestamp": "2023-05-11T21:21:22.763132720Z",
  "spanId": "583ecd81fa3bcc49"
}

排查与解决方法 #

1. 启用gRPC服务的NEG配置 #

nginx-grpc Service缺少cloud.google.com/neg: '{"ingress": true}'注解。GKE Ingress对接HTTP/2服务必须启用Network Endpoint Groups (NEG)，否则负载均衡无法正确路由到后端Pod。

修改后的nginx-grpc Service配置：

apiVersion: v1
kind: Service
metadata:
  namespace: drm
  name: nginx-grpc
  annotations:
    cloud.google.com/backend-config: '{"default":"http-hc-config"}'
    cloud.google.com/app-protocols: '{"grpc":"HTTP2"}'
    cloud.google.com/neg: '{"ingress": true}'  # 添加该注解
spec:
  ports:
    - protocol: TCP
      port: 9900
      targetPort: 9900
      name: grpc
    - protocol: TCP
      port: 2222
      targetPort: 2222
      name: health
  selector:
    app: nginx
  type: NodePort

2. 优化BackendConfig与服务端口的关联 #

将BackendConfig的健康检查端口改为使用服务中定义的端口名，避免端口号歧义：

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  namespace: drm
  name: http-hc-config
spec:
  healthCheck:
    checkIntervalSec: 60
    requestPath: /health
    type: HTTP
    portName: health  # 使用服务端口名替代直接端口号

同时更新Service的BackendConfig注解，显式绑定各端口的健康检查配置：

cloud.google.com/backend-config: '{"ports": {"health":"http-hc-config", "grpc":"http-hc-config"}}'

3. 验证Nginx的HTTP2监听配置 #

进入Nginx Pod验证配置是否生效：

kubectl exec -n drm <nginx-pod-name> -- nginx -T | grep -A10 "listen 9900"

确认输出包含http2关键字，确保Nginx在9900端口开启HTTP2监听。

4. 检查Pod健康状态 #

确认Nginx Pod的就绪探针和存活探针正常，Pod处于Ready状态：

kubectl get pods -n drm

若探针失败，负载均衡会将Pod从后端池中移除，导致无法连接。

5. 验证后端gRPC服务可达性 #

在Nginx Pod内部测试communication-terminal-grpc:9903的连通性：

kubectl exec -n drm <nginx-pod-name> -- grpcurl communication-terminal-grpc:9903 list

如果此命令失败，需先解决后端gRPC服务的运行问题。

内容的提问来源于stack exchange，提问作者ANTONIO LOPEZ