Azure Kubernetes(AKS)中间歇性502 Bad Gateway错误排查求助

问题背景 #

• 近期将应用从VM迁移至Azure Kubernetes Service（AKS）环境，前端基于React开发，后端包含11-12个Node.js微服务
• 仅其中一个负载较高的微服务（对应lever-lender-service）偶发随机502 Bad Gateway错误
• 查看该微服务的Kubernetes Pod日志，未获取到有效排查信息

已尝试的排查/缓解措施 #

• 怀疑自动扩缩容创建的新Pod在实际就绪前被调度器标记为可用，已将该微服务的最小Pod数增加至10，问题仍存在
• 查看Nginx Ingress Controller日志，仅能看到502错误记录，无有效调试细节

Ingress日志示例 #

24.171.57.159 - - [08/Dec/2022:21:40:27 +0000] "POST /lender/api/v1/dealer/getUserDetailsById HTTP/2.0" 502 150 "https://portal.leverauto.com/Admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15" 402 51.967 [default-lever-lender-service-5009] [] 10.240.0.155:5009 0 51.935 502 b8a41fdba7930aea0a9b22780024bd65

Ingress配置文件 #

ingress-r.yaml（前端路由，无需路径重写） #

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-r
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "360"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "360"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "360"
    nginx.ingress.kubernetes.io/affinity: cookie
    
spec:
  tls:
  - hosts:
    - portal.leverauto.com
    secretName: lever-ssl
  defaultBackend:
    service:
      name: lever-webapp-service
      port:
        number: 80
  rules:
  - host: portal.leverauto.com
    http: &http_rules
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: lever-webapp-service
            port:
              number: 80 

ingress-srv.yaml（后端微服务，含路径重写） #

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-srv
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "360"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "360"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "360"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/affinity: cookie

spec:
  tls:
  - hosts:
    - portal.leverauto.com
    secretName: lever-ssl
  defaultBackend:
    service:
      name: lever-webapp-service
      port:
        number: 80
  rules:
  - host: portal.leverauto.com
    http: &http_rules
      paths:
      - path: /auth(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: lever-authentication-service
            port:
              number: 5000
      - path: /auction(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: lever-auction-service
            port:
              number: 5010 
      - path: /audit(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: lever-audits-service
            port:
              number: 5005                      
      - path: /inventory(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: lever-inventory-service
            port:
              number: 5006         
      - path: /lender(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: lever-lender-service
            port:
              number: 5009 
      - path: /payment(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: lever-payments-service
            port:
              number: 5002
      - path: /report(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: lever-reports-service
            port:
              number: 5003 
      - path: /sopraData(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: lever-sopra-datapull-service
            port:
              number: 5011
      - path: /sopra(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: lever-sopra-service
            port:
              number: 5008
      - path: /vehicle(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: lever-vehicle-service
            port:
              number: 5001
      - path: /workflow(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: lever-workflow-service
            port:
              number: 5004 

内容的提问来源于stack exchange，提问作者Harshith