EKS集群中调用GRPC端点时Envoy Sidecar报connection_termination错误求助

问题描述 #

应用部署在EKS集群，8080端口提供健康检查端点（可正常访问），8888端口暴露GRPC端点。执行命令：

grpcurl -plaintext eks-test.my-app.abc.com:8888 list

时，Envoy Sidecar日志出现连接终止错误，关键日志内容如下：

{
    "x_forwarded_for":"10.164.21.112",
    "protocol":"HTTP/1.1",
    "authority":"eks-test.my-abc.com:8888",
    "start_time":"2023-06-21T20:15:50.453Z",
    "response_flags":"UC",
    "response_code_details":"upstream_reset_before_response_started{connection_termination}",
    "connection_termination_details":null,
    "upstream_host":"10.164.44.90:8080",
    "upstream_cluster":"inbound|8080||",
    "upstream_local_address":"127.0.0.6:35681",
    "requested_server_name":"outbound_.8080_._.my-app.gom.svc.cluster.local",
    "method":"POST",
    "request_id":"7e68bac0-83cb-9fd6-8b38-554dd52f9ea9",
    "upstream_transport_failure_reason":null,
    "response_code":200,
    "downstream_local_address":"10.164.44.90:8080",
    "downstream_remote_address":"10.164.21.112:0",
    "path":"/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo",
    "upstream_service_time":null,
    "user_agent":"grpcurl/1.8.7 grpc-go/1.48.0",
    "route_name":"default",
    "bytes_sent":0
}

当前资源配置 #

Deployment #

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations: {}
  labels:
    app: my-app
  name: my-app
  namespace: gom
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: nodegroup-name
                    operator: In
                    values:
                      - general
      containers:
        - envFrom:
            - configMapRef:
                name: package-orchestrator-app-env
          image: 'docker.repo.frg.tech/gom/my-app:0.0.1'
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 150
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: rpc
            timeoutSeconds: 1
          name: my-app
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
            - containerPort: 8888
              name: rpc
              protocol: TCP
          resources:
            limits:
              cpu: 4
              memory: 4Gi
            requests:
              cpu: 2
              memory: 1Gi
      serviceAccountName: my-app

Service #

apiVersion: v1
kind: Service
metadata:
  annotations: {}
  labels:
    app: my-app
  name: my-app
  namespace: gom
spec:
  ports:
    - name: http
      port: 8080
      targetPort: 8080
    - name: grpc-rpc
      port: 8888
      targetPort: 8888
      appProtocol: http2
  selector:
    app: my-app
  type: ClusterIP

Gateway #

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  labels:
    app: my-app
  name: my-app
  namespace: gom
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - eks-test.my-app.abc.com
      port:
        name: http
        number: 8080
        protocol: HTTP
      tls:
        httpsRedirect: true
    - hosts:
        - eks-test.my-app.abc.com
      port:
        name: https
        number: 8443
        protocol: HTTPS
      tls:
        credentialName: my-app
        mode: SIMPLE
    - hosts:
        - eks-test.my-app.abc.com
      port:
        name: grpc
        number: 8888
        protocol: GRPC

VirtualService #

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  labels:
    app: my-app
  name: my-app
  namespace: gom
spec:
  gateways:
    - my-app
  hosts:
    - eks-test.my-app.abc.com
  http:
    - match:
        - uri:
            prefix: /
      route:
        - destination:
            host: my-app.gom.svc.cluster.local
            port:
              number: 8080
    - match:
        - uri:
            prefix: /
      route:
        - destination:
            host: my-app.gom.svc.cluster.local
            port:
              number: 8888

问题分析 #

1. 路由规则冲突：VirtualService中两条HTTP路由的匹配条件都是uri: prefix: /，Istio会优先匹配第一条规则，导致所有请求（包括GRPC）都被转发到8080端口，和日志中upstream_host: 10.164.44.90:8080的信息一致。
2. 协议不匹配：GRPC基于HTTP/2协议，被错误路由到处理HTTP/1.1的8080端口，导致连接被上游终止，触发upstream_reset_before_response_started{connection_termination}错误。

修复方案 #

1. 修改VirtualService，区分HTTP与GRPC流量 #

通过请求头content-type: application/grpc识别GRPC流量，优先转发到8888端口，修改后的配置如下：

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  labels:
    app: my-app
  name: my-app
  namespace: gom
spec:
  gateways:
    - my-app
  hosts:
    - eks-test.my-app.abc.com
  http:
    # 优先处理GRPC流量
    - match:
        - headers:
            content-type:
              exact: application/grpc
      route:
        - destination:
            host: my-app.gom.svc.cluster.local
            port:
              number: 8888
    # 处理普通HTTP流量
    - match:
        - uri:
            prefix: /
      route:
        - destination:
            host: my-app.gom.svc.cluster.local
            port:
              number: 8080

2. 优化Gateway的GRPC端口配置（可选） #

为Gateway的GRPC端口添加appProtocol: http2，和Service配置保持一致，明确协议类型：

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  labels:
    app: my-app
  name: my-app
  namespace: gom
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - eks-test.my-app.abc.com
      port:
        name: http
        number: 8080
        protocol: HTTP
      tls:
        httpsRedirect: true
    - hosts:
        - eks-test.my-app.abc.com
      port:
        name: https
        number: 8443
        protocol: HTTPS
      tls:
        credentialName: my-app
        mode: SIMPLE
    - hosts:
        - eks-test.my-app.abc.com
      port:
        name: grpc
        number: 8888
        protocol: GRPC
        appProtocol: http2

3. 验证修复 #

应用配置修改后，重新执行grpcurl -plaintext eks-test.my-app.abc.com:8888 list，检查Envoy日志是否将请求路由到8888端口，确认GRPC服务可正常访问。

内容的提问来源于stack exchange，提问作者esahmo