GKE集群中Nginx Ingress返回503错误的排查求助

环境与部署步骤 #

我正在学习Kubernetes，基于GKE搭建了一套环境，执行以下步骤部署组件：

1. 安装Nginx Ingress #

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.0/deploy/static/provider/cloud/deploy.yaml

2. 部署Ingress资源 #

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "*"
    nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, GET, POST, OPTIONS, DELETE"
    nginx.ingress.kubernetes.io/cors-allow-headers: "DNT,X-CustomHeader,X-LANG,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,X-Api-Key,X-Device-Id,Access-Control-Allow-Origin"
  
spec:
  rules:
  - http:
      paths:
      - path: /?(.*)
        pathType: Prefix
        backend:
          service:
            name: client-service
            port:
              number: 3000
      - path: /api/?(.*)
        pathType: Prefix
        backend:
          service:
            name: server-service
            port:
              number: 5000
  ingressClassName: nginx

3. 部署前端服务与Deployment #

Service

apiVersion: v1
kind: Service
metadata:
  name: client-service
spec:
  ports:
    - port: 3000
      protocol: TCP
      targetPort: http-port
  selector:
    app: client
  type: ClusterIP

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: client-deployment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: client
    spec:
      containers:
        - name: client
          image: jowz/client:v4.2
          ports:
            - containerPort: 3000
              name: http-port
  selector:
    matchLabels:
      app: client

4. 部署后端服务与Deployment #

Service

apiVersion: v1
kind: Service
metadata:
  name: server-service
spec:
  ports:
    - port: 5000
      protocol: TCP
  selector:
    app: server
  type: ClusterIP

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: server-deployment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: server
    spec:
      containers:
        - name: server
          image: jowz/server:v1
          imagePullPolicy: Always
          ports:
            - containerPort: 5000
              name: nodejs-port
  selector:
    matchLabels:
      app: server

问题现象 #

访问Ingress时持续返回503错误。已检查服务名称、Selector匹配，所有Pod均处于Running状态，Pod日志无错误。执行kubectl get pods --all-namespaces输出如下：

NAMESPACE       NAME                                                 READY   STATUS      RESTARTS   AGE
ingress-nginx   client-deployment-9ccf8cf87-7g6qr                    1/1     Running     0          40h
ingress-nginx   ingress-nginx-admission-create-tsp4n                 0/1     Completed   0          44h
ingress-nginx   ingress-nginx-admission-patch-q7spm                  0/1     Completed   1          44h
ingress-nginx   ingress-nginx-controller-86b55bb769-2bqs5            1/1     Running     0          44h
ingress-nginx   mongodb                                              1/1     Running     0          41h
ingress-nginx   server-deployment-df679664f-99nrq                    1/1     Running     0          41h
kube-system     event-exporter-gke-d4b7ff94-4st8l                    2/2     Running     0          2d17h
kube-system     fluentbit-gke-xlv7v                                  2/2     Running     0          2d17h
kube-system     gke-metrics-agent-fhnmq                              2/2     Running     0          2d17h
kube-system     konnectivity-agent-697c66b96-bvfdw                   1/1     Running     0          2d17h
kube-system     konnectivity-agent-autoscaler-864fff96c4-n9tlp       1/1     Running     0          2d17h
kube-system     kube-dns-autoscaler-758c4689b9-7gzx8                 1/1     Running     0          2d17h
kube-system     kube-dns-fc686db9b-hjl4d                             4/4     Running     0          2d17h
kube-system     kube-proxy-gke-octodemo-default-pool-20afc590-94qj   1/1     Running     0          2d17h
kube-system     l7-default-backend-9db4bd868-zgx8s                   1/1     Running     0          2d17h
kube-system     metrics-server-6f6cdbf67d-m6hgv                      1/1     Running     0          40h
kube-system     metrics-server-v0.5.2-66bbcdbffc-bmzp6               2/2     Running     0          2d17h
kube-system     pdcsi-node-st68b                                     2/2     Running     0          40h

集群托管在GKE上，控制台无错误提示，不清楚还能通过哪些方式调试该问题。

调试建议 #

• 检查命名空间一致性：从Pod列表看，client/server Pod都在ingress-nginx命名空间，确认Ingress资源是否也部署在该命名空间。如果Ingress在默认命名空间，需要在服务名称后添加命名空间后缀（如client-service.ingress-nginx.svc.cluster.local），或者将Ingress移到ingress-nginx命名空间。
• 验证Service Endpoints：执行以下命令检查Service是否关联了Pod的IP：

  kubectl get endpoints client-service -n ingress-nginx
  kubectl get endpoints server-service -n ingress-nginx
  

  如果Endpoints为空，重新核对Service的Selector和Pod的标签是否完全匹配（可通过kubectl describe pod <pod-name> -n ingress-nginx和kubectl describe service <service-name> -n ingress-nginx查看细节）。
• 测试服务连通性：进入Ingress Controller Pod内部，直接curl服务地址，验证是否能正常访问：

  kubectl exec -it <ingress-controller-pod-name> -n ingress-nginx -- curl client-service.ingress-nginx.svc.cluster.local:3000
  kubectl exec -it <ingress-controller-pod-name> -n ingress-nginx -- curl server-service.ingress-nginx.svc.cluster.local:5000
  

• 查看Ingress Controller日志：检查Controller的转发日志，是否有连接失败、DNS解析错误等信息：

  kubectl logs <ingress-controller-pod-name> -n ingress-nginx
  

• 检查GKE负载均衡状态：在GKE控制台查看Ingress对应的负载均衡器，确认后端服务的健康检查是否通过，节点池关联是否正确。
• 确认IngressClass配置：执行kubectl get ingressclass，检查是否存在名为nginx的IngressClass，且其关联的Controller是Nginx Ingress Controller。

内容的提问来源于stack exchange，提问作者Jowz