Kubernetes部署Kibana遇Nginx Ingress 502 Bad Gateway问题

问题背景 #

在Kubernetes集群部署Kibana后，本地通过port-forward可正常访问，但配置Ingress后出现502 Bad Gateway错误，证书及Secret配置正常。

相关配置及日志信息 #

现有Ingress YAML配置 #

kind: Ingress
metadata:
  name: kibana
  namespace: quickstart
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
  ingressClassName: nginx
  rules:
    - host: kibana.mydomain.com
      http:
        paths:
        - backend:
            service:
              name: quickstart-kb-http 
              port:
                number: 5601
          path: /
          pathType: Prefix
  tls:
  - hosts:
    - kibana.mydomain.com
    secretName: mydomain-secret
status:
  loadBalancer:
    ingress:
    - ip: 10.55.55.55

Ingress日志 #

2023/03/07 10:19:56 [error] 11977#11977: *278798777 upstream prematurely closed connection while reading response header from upstream, client: 10.41.41.41, server: kibana.mydomain.com, request: "GET /favicon.ico HTTP/2.0", upstream: "http://172.2.2.4:5601/favicon.ico", host: "kibana.mydomain.com", referrer: "https://kibana.mydomain.com/"
2023/03/07 10:19:56 [error] 11977#11977: *278798777 upstream prematurely closed connection while reading response header from upstream, client: 10.41.41.41, server: kibana.mydomain.com, request: "GET /favicon.ico HTTP/2.0", upstream: "http://172.2.2.4:5601/favicon.ico", host: "kibana.mydomain.com", referrer: "https://kibana.mydomain.com/" 10.41.41.41 - - [07/Mar/2023:10:19:56 +0000] "GET /favicon.ico HTTP/2.0" 502 552 "https://kibana.mydomain.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)Chrome/110.0.0.0 Safari/537.36" 146 0.018 [quickstart-quickstart-kb-http-5601] [] 172.2.2.4:5601, 172.2.2.4:5601, 172.2.2.4:5601 0, 0, 0 0.012, 0.004, 0.004 502, 502, 502 81a10e31ca6778cbfd738bae3195dabf

Service信息 #

DESKTOP:~$ kubectl get services quickstart-kb-http -n quickstart
NAME              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
quickstart-kb-http   ClusterIP   172.2.2.4   <none>        5601/TCP   1d1h

Pod信息 #

DESKTOP:~$ kubectl get pod quickstart-kb-5261adb64f-wtq41 -n quickstart
NAME                          READY   STATUS    RESTARTS   AGE
quickstart-kb-5261adb64f-wtq41   1/1     Running   0          20h

本地端口转发命令 #

kubectl port-forward quickstart-kb-5261adb64f-wtq41 5601:5601

补充Ingress信息 #

@DESKTOP:~$ kubectl get ingress -n quickstart
NAME     CLASS   HOSTS                ADDRESS         PORTS     
kibana   nginx   kibana.mydomain.com   10.55.55.55   80, 443

环境信息 #

• OpenSSL版本：OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

排查建议 #

1. 验证Service到Pod的连通性 #

在集群内任意Pod（比如Ingress Controller的Pod）中执行以下命令，测试Service的5601端口是否可达：

curl -v http://172.2.2.4:5601

若连接被拒绝或无响应，需检查：

• Service的selector是否与Pod的labels匹配：执行kubectl describe service quickstart-kb-http -n quickstart查看selector，对比Pod标签（kubectl get pod quickstart-kb-5261adb64f-wtq41 -n quickstart -o jsonpath='{.metadata.labels}'）
• Pod的5601端口是否监听在0.0.0.0：进入Pod执行netstat -tulpn | grep 5601，Kibana默认可能绑定localhost，需修改配置server.host: "0.0.0.0"

2. 检查Kibana的Server配置 #

确认Kibana配置中是否设置了以下参数：

server.host: "0.0.0.0"
server.publicBaseUrl: "https://kibana.mydomain.com/"

如果是Helm部署，需在values.yaml中设置后重新部署。

3. 调整Ingress超时配置 #

日志中出现upstream prematurely closed connection，可能是超时导致，添加以下Ingress注解：

nginx.ingress.kubernetes.io/proxy-connect-timeout: "60"
nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
nginx.ingress.kubernetes.io/proxy-send-timeout: "60"

4. 检查网络策略限制 #

若集群启用了网络策略，确认是否允许Ingress Controller所在Namespace的流量访问quickstart Namespace的5601端口。

5. 验证Kibana请求来源限制 #

检查Kibana配置中的server.allowedHosts参数，确保包含kibana.mydomain.com，避免请求被拦截。

内容的提问来源于stack exchange，提问作者Mightiev