GKE中ingress-nginx-controller与istio-ingressgateway通信502问题

Kubernetes集群中，ingress-nginx-controller作为LoadBalancer，对Istio服务网格外的应用访问正常，但访问网格内应用时持续返回502 Bad Gateway错误。期望的流量路径为：
[ingress-nginx-controller（LoadBalancer）] ===> [Istio Gateway] ===> [K8s Service]

尝试参考AWS ALB前置Istio Gateway的教程但未解决问题，以下是相关配置：

Ingress配置 #

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: istio-ingress
  namespace: istio-ingress
  annotations:
    certmanager.k8s.io/acme-challenge-type: dns01
    certmanager.k8s.io/cluster-issuer: letsencrypt
    # nginx.ingress.kubernetes.io/service-upstream: "true"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - "*.example.com"
        - "example.com"
      secretName: wildcard-istio-cert
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: istio-ingressgateway
                port:
                   number: 443 # 尝试后返回502
                  # name: http-web 尝试后完全无法访问
                  # name: https-ssl 尝试后返回502
                  # number: 30824 对应https-ssl的NodePort，返回503
                  # number: 31849 对应http-web的NodePort，返回503
                  # number: 30276 对应status-port的NodePort，返回503

Istio IngressGateway（NodePort类型Service）配置 #

kind: Service
apiVersion: v1
metadata:
  name: istio-ingressgateway
  namespace: istio-ingress
  uid: b6d1f4ce-2af4-42f0-be10-9e5d32edc716
  resourceVersion: "58534080"
  creationTimestamp: "2024-02-01T19:39:32Z"
  labels:
    app: istio-ingressgateway
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: istio-ingressgateway
    app.kubernetes.io/version: 1.20.0
    helm.sh/chart: gateway-1.20.0
    istio: ingressgateway
  annotations:
    cloud.google.com/neg: '{"ingress":true}'
    meta.helm.sh/release-name: istio-ingressgateway
    meta.helm.sh/release-namespace: istio-ingress
spec:
  ports:
  - name: status-port
    protocol: TCP
    port: 15021
    targetPort: 15021
    nodePort: 30276
  - name: http-web
    protocol: TCP
    port: 80
    targetPort: 80
    nodePort: 31849
  - name: https-ssl
    protocol: TCP
    port: 443
    targetPort: 443
    nodePort: 30824
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  clusterIP: 34.118.227.128
  clusterIPs:
  - 34.118.227.128
  type: NodePort
  sessionAffinity: None
  externalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  internalTrafficPolicy: Cluster
status:
  loadBalancer: {}

ingress-nginx-controller（LoadBalancer类型Service）配置 #

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
  resourceVersion: "51046251"
  creationTimestamp: "2024-01-25T22:40:52Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.9.0
    helm.sh/chart: ingress-nginx-4.8.0
  annotations:
    cloud.google.com/neg: '{"ingress":true}'
    meta.helm.sh/release-name: ingress-nginx
    meta.helm.sh/release-namespace: ingress-nginx
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
spec:
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: http
    nodePort: 32638
  - name: https
    protocol: TCP
    port: 443
    targetPort: https
    nodePort: 32450
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  clusterIP: {MY-CLUSTER-IP}
  clusterIPs:
  - 34.118.227.191
  type: LoadBalancer
  sessionAffinity: None
  loadBalancerIP: {MY-IP}
  externalTrafficPolicy: Local
  healthCheckNodePort: 31936
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  allocateLoadBalancerNodePorts: true
  internalTrafficPolicy: Cluster
status:
  loadBalancer:
    ingress:
    - ip: {MY-IP}

核心问题定位 #

当前配置的核心矛盾是：ingress-nginx已完成TLS终止（配置了tls字段并引用证书），但又将流量转发给Istio IngressGateway的443端口（HTTPS），导致流量被重复加密/解密，或Istio Gateway无法正确处理来自ingress-nginx的请求。

具体修复步骤 #

1. 调整Ingress转发端口与注解
   修改Ingress配置，将后端端口指向Istio IngressGateway的HTTP端口（80），并启用nginx.ingress.kubernetes.io/service-upstream: "true"注解，让ingress-nginx直接转发到Istio Gateway的ClusterIP，避免NodePort的额外网络跳转：

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: istio-ingress
     namespace: istio-ingress
     annotations:
       certmanager.k8s.io/acme-challenge-type: dns01
       certmanager.k8s.io/cluster-issuer: letsencrypt
       nginx.ingress.kubernetes.io/service-upstream: "true" # 启用该注解
   spec:
     ingressClassName: nginx
     tls:
       - hosts:
           - "*.example.com"
           - "example.com"
         secretName: wildcard-istio-cert
     rules:
       - host: app.example.com
         http:
           paths:
             - path: /
               pathType: Prefix
               backend:
                 service:
                   name: istio-ingressgateway
                   port:
                     number: 80 # 改为HTTP端口
   

2. 确认Istio Gateway与VirtualService配置
   检查Istio Gateway资源是否在80端口开启HTTP监听器，并绑定对应域名：

   apiVersion: networking.istio.io/v1alpha3
   kind: Gateway
   metadata:
     name: istio-ingressgateway
     namespace: istio-ingress
   spec:
     selector:
       istio: ingressgateway
     servers:
     - port:
         number: 80
         name: http
         protocol: HTTP
       hosts:
       - "app.example.com"
   

   同时确保VirtualService已正确关联Gateway和后端服务：

   apiVersion: networking.istio.io/v1alpha3
   kind: VirtualService
   metadata:
     name: app-vs
     namespace: your-app-namespace
   spec:
     hosts:
     - "app.example.com"
     gateways:
     - istio-ingress/istio-ingressgateway # 对应Gateway的命名空间与名称
     http:
     - route:
       - destination:
           host: your-app-service-name
           port:
             number: 8080 # 应用服务的端口
   

3. 链路验证

   • 测试Istio Gateway内部访问：

     kubectl run -it --rm --image=curlimages/curl curl-test -- curl http://istio-ingressgateway.istio-ingress.svc.cluster.local -H "Host: app.example.com"
     

     返回正常则说明Istio内部链路无问题。
   • 测试ingress-nginx转发：

     kubectl run -it --rm --image=curlimages/curl curl-test -- curl https://app.example.com --resolve app.example.com:443:{INGRESS-NGINX-LOADBALANCER-IP}
     

4. 日志排查

   • 查看Istio IngressGateway Pod日志，检查请求转发错误：

     kubectl logs -n istio-ingress -l app=istio-ingressgateway
     

   • 查看ingress-nginx-controller Pod日志，检查连接Istio Gateway的失败记录：

     kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx
     

内容的提问来源于stack exchange，提问作者CommonSenseCode