GCP环境Nginx Ingress 504超时问题排查与配置方案

问题背景 #

• 部署在GCP上的Spring Boot应用内置Admin账号管理面板，其中账号迁移接口需要执行数据备份、旧服务删除、基于新镜像重建服务等逻辑，耗时可达60秒以上
• 本地运行应用直连GCP环境时，该接口可正常执行完成；代码合并到master分支完成GCP环境构建部署后，调用接口返回504 Gateway Time-out错误，报错截图：
  
• 已排除Java应用自身代码故障，初步判断超时来自负载均衡层

采集到的相关报错 #

ingress-nginx控制器错误日志 #

2022-06-17T09:17:48.165983390Z2022/06/17 09:17:48 [error] 1922#1922: *1134692 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 80.192.45.186, server: _, request: "POST /admin/accounts/123456/edit HTTP/1.1", upstream: "http://XX.XX.XX.XXX:8080/accounts/123456/edit", host: "somehost.com", referrer: "https://somehost.com/admin/accounts/123456/edit"

GCP Logs Explorer结构化日志 #

{
  "textPayload": "2022/06/17 09:17:48 [error] 1922#1922: *1134692 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 80.192.45.186, server: _, request: \"POST /admin/accounts/123456/edit HTTP/1.1\", upstream: \"http://XX.XX.XX.XXX:8080/accounts/123456/edit\", host: \"somehost.com\", referrer: \"https://somehost.com/admin/accounts/123456/edit\"",
  "insertId": "03uw65hlnarbu7v1",
  "resource": {
    "type": "k8s_container",
    "labels": {
      "namespace_name": "ingress-nginx",
      "container_name": "controller",
      "project_id": "my_project",
      "cluster_name": "my_cluster",
      "pod_name": "ingress-nginx-controller-cb576846d-6xxvf",
      "location": "europe-west1"
    }
  },
  "timestamp": "2022-06-17T09:17:48.165983390Z",
  "severity": "ERROR",
  "labels": {
    "k8s-pod/pod-template-hash": "cb576846d",
    "k8s-pod/app_kubernetes_io/instance": "ingress-nginx",
    "compute.googleapis.com/resource_name": "my_resource",
    "k8s-pod/app_kubernetes_io/name": "ingress-nginx",
    "k8s-pod/app_kubernetes_io/component": "controller"
  },
  "logName": "projects/my_project/logs/stderr",
  "receiveTimestamp": "2022-06-17T09:17:50.861580453Z"
}

现有部署配置 #

此前已尝试在Terraform脚本、GCP控制台调整超时参数均未生效，当前部署ingress-controller使用的Terraform配置如下：

resource "kubernetes_ingress" "nginx-ingress" {
  metadata {
    name      = "nginx-ingress"
    namespace = kubernetes_namespace.nginx.metadata[0].name
    annotations = {
      "kubernetes.io/ingress.class"                 = "gce"
      "kubernetes.io/ingress.global-static-ip-name" = "ext-lb"
      "ingress.gcp.kubernetes.io/pre-shared-cert"   = local.ssl_cert_name
      "networking.gke.io/v1beta1.FrontendConfig"    = kubernetes_manifest.nginx_frontend_config.manifest["metadata"]["name"]
    }
  }
  spec {
    rule {
      host = local.domain_name
      http {
        path {
          backend {
            service_name = "ingress-nginx-controller"
            service_port = 80
          }
        }
        path {
          path = "/secured/*"
          backend {
            service_name = "ingress-iap-nginx-controller"
            service_port = 80
          }
        }
      }
    }
  }
}

根因说明 #

当前集群采用两层负载均衡架构，之前配置不生效的核心原因是配置位置错误、未同时覆盖两层LB的超时阈值：

1. 外层是GCP原生HTTP(S)负载均衡，由配置中ingress.class=gce的Ingress资源创建，默认请求超时为30秒
2. 内层是自行部署的Nginx Ingress Controller，默认向后端服务读取响应的超时为60秒
3. 账号迁移接口耗时超过60秒时，任意一层LB达到超时阈值都会中断请求返回504。从日志可以明确，当前是内层Nginx Ingress先触发了upstream读响应超时：之前仅调整了外层GCP LB的配置，且关联的FrontendConfig未正确配置超时参数，完全没有调整内层Nginx的转发超时规则，因此所有配置修改都不生效。

正确配置步骤 #

需要同时调整两层负载均衡的超时阈值，将超时设置为大于接口最大耗时的值（示例设置为120秒，可根据实际业务耗时调整）。

1. 配置外层GCP负载均衡超时 #

在引用的nginx_frontend_config资源中增加timeout配置块：

resource "kubernetes_manifest" "nginx_frontend_config" {
  manifest = {
    "apiVersion" = "networking.gke.io/v1beta1"
    "kind"       = "FrontendConfig"
    "metadata" = {
      "name"      = "nginx-frontend-config"
      "namespace" = kubernetes_namespace.nginx.metadata[0].name
    }
    "spec" = {
      # 新增超时配置，单位为秒
      "timeout" = {
        "timeoutSec" = 120
      }
      # 原有其他配置（如SSL重定向、IAP规则）保持不变
    }
  }
}

2. 配置内层Nginx Ingress转发超时 #

注意：Nginx的超时注解不要加在外层GCE类的Ingress资源上，加了也不会被识别，需要加在业务服务对应的Nginx类Ingress资源上

# Spring Boot业务服务对应的Nginx Ingress资源
resource "kubernetes_ingress_v1" "springboot_app_ingress" {
  metadata {
    name      = "springboot-app-ingress"
    namespace = "业务服务所在命名空间"
    annotations = {
      "kubernetes.io/ingress.class" = "nginx"
      # 配置Nginx向后端服务的连接、读、写超时
      "nginx.ingress.kubernetes.io/proxy-connect-timeout" = "10"
      "nginx.ingress.kubernetes.io/proxy-read-timeout" = "120"
      "nginx.ingress.kubernetes.io/proxy-send-timeout" = "120"
    }
  }
  # 原有业务路由规则保持不变
  spec {
    # 路由规则省略
  }
}

如果所有业务接口都需要统一长超时配置，可以直接全局修改Nginx Ingress Controller的ConfigMap，无需逐个Ingress加注解：

resource "kubernetes_config_map_v1" "nginx_ingress_controller_config" {
  metadata {
    name      = "ingress-nginx-controller"
    namespace = kubernetes_namespace.nginx.metadata[0].name
  }
  data = {
    "proxy-connect-timeout" = "10"
    "proxy-read-timeout" = "120"
    "proxy-send-timeout" = "120"
  }
}

配置应用完成后等待1-2分钟让LB规则生效，再调用接口验证即可。

架构优化建议 #

对于耗时超过30秒的重操作，不建议使用同步接口阻塞等待，最好改成异步任务模式：接口收到迁移请求后立即返回任务ID，后台异步执行迁移逻辑，前端通过轮询任务进度接口查询执行状态，从架构层面避免网关超时问题。

内容的提问来源于stack exchange，提问作者Artet_dev