EKS中grpc-dotnet服务配置NLB TLS终止遇连接失败问题

问题场景 #

在EKS集群中，使用AWS Load Balancer Controller为grpc-dotnet实现的gRPC服务配置带TLS终止的内部NLB，NLB与Pod之间采用明文TCP流量。对应的Kubernetes Service配置如下：

apiVersion: v1
kind: Service
metadata:
    annotations:
        service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: /health
        service.beta.kubernetes.io/aws-load-balancer-internal: 'true'
        service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
        service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: "ELBSecurityPolicy-TLS13-1-2-2021-06"
        service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-west-1:account:certificate/my-cert-arn"
        service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "9090"
        # service.beta.kubernetes.io/aws-load-balancer-alpn-policy: HTTP2Preferred
    name: grpc-api-svc
    namespace: default
spec:
    ports:
        - 
            port: 9090
            protocol: TCP
            targetPort: 9090
    selector:
        app: grpc-api
    type: LoadBalancer

配置后流量可从NLB路由到gRPC API，但请求会报错；移除SSL相关配置后服务恢复正常，添加ALPN策略HTTP2Preferred也无法解决问题。另外发现Python实现的gRPC服务可正常工作，仅grpc-dotnet服务出现该问题。

错误信息 #

{
  "created": "@1678456920.910000000",
  "description": "Failed to pick subchannel",
  "file": "src/core/ext/filters/client_channel/client_channel.cc",
  "file_line": 5391,
  "referenced_errors": [
    {
      "created": "@1678456920.909000000",
      "description": "failed to connect to all addresses",
      "file": "src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc",
      "file_line": 398,
      "grpc_status": 14
    }
  ]
}

根本原因 #

NLB终止TLS后，会保留请求中的https协议头，但Kestrel服务器默认对协议头的校验较为严格，会直接忽略这类请求。

解决方案 #

在grpc-dotnet服务的Kestrel配置中开启AllowAlternateSchemes选项，允许服务器接受带有非本地监听协议的请求头：

builder.WebHost.ConfigureKestrel(options =>
{
    options.AllowAlternateSchemes = true;
});

内容的提问来源于stack exchange，提问作者Holden Cauldfield