使用AWS ELB时Istio滚动更新出现间歇性504错误

问题描述 #

我们的Istio 1.12.6部署已稳定运行一段时间，但重启Ingress Gateway Pod时，流量中会随机出现间歇性504错误，同时存在正常请求。

推测该问题由ELB与IGW Pod间的Keepalive连接导致，依据如下：

• 网关侧无504错误日志
• LB指标显示的是ELB 5xx错误，而非HTTP 5xx

已尝试的操作 #

• 将terminationDrainDuration配置为70s（长于LB的60s空闲超时时间），Pod确实按预期时长关闭，但问题未解决
• 复现方法：持续向应用发送流量，执行kubectl rollout restart deployment/my-ingressgateway命令重启IGW Pod

当前配置 #

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: my-iop
  namespace: my-ns
spec:
  meshConfig:
    defaultConfig:
      # For some reason this has no effect on the IGW pods?
      terminationDrainDuration: 70s
      holdApplicationUntilProxyStarts: true
      proxyMetadata:
        ISTIO_META_EXIT_ON_ZERO_ACTIVE_CONNECTIONS: "true"
  components:
    cni:
      enabled: true
    pilot:
      enabled: true
    ingressGateways:
      - enabled: true
        k8s:
          overlays:
            - kind: Deployment
              name: my-ingressgateway
              patches:
                - path: spec.template.spec.terminationGracePeriodSeconds
                  value: 120
          podAnnotations:
            # setting this does actually affect the IGW pods, the logs print the configuration correctly
            proxy.istio.io/config: |
              terminationDrainDuration: 70s
              proxyMetadata:
                ISTIO_META_EXIT_ON_ZERO_ACTIVE_CONNECTIONS: "true"
          hpaSpec:
            minReplicas: 5
            maxReplicas: 10
          serviceAnnotations:
            service.beta.kubernetes.io/aws-load-backend-protocol: https
            service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
            service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: TCP
            service.beta.kubernetes.io/aws-load-balancer-healthcheck-target: TCP
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"
            service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
            service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "<CERT-ARN>" # TLS termination at the LB, re-encrypt between LB and IGW pods
            service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
            service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60" # This is also the default
        name: my-ingressgateway
  profile: minimal
  tag: 1.12.6-distroless
---
apiVersion: networking.istio.io/v1alpha1
kind: Gateway
metadata:
  name: my-gateway
  namespace: my-ns
spec:
  selector:
    istio: my-ingressgateway
  servers:
    - hosts:
        - "*"
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        credentialName: the-credential
        mode: SIMPLE
---
apiVersion: networking.istio.io/v1alpha1
kind: VirtualService
metadata:
  name: my-vs
  namespace: app-ns
spec:
  gateways:
    - my-ns/my-gateawy
  hosts:
    - my-cool-hostname.example.com
  http:
    - match:
        - uri:
          regex: .*/invoke
      rewrite:
        uri: /invoke
      route:
        - destination:
            host: the-app
            port:
              number: 9000

恳请各位提供技术解决方案或排查思路！

内容的提问来源于stack exchange，提问作者PSU2017