通过Helm与nginx-ingress暴露MinIO控制台遇HTTP 400问题求助

问题现象 #

• 在GCP GKE环境中，使用MinIO Operator、Tenant、Helm Chart结合ArgoCD完成部署，通过端口转发访问MinIO Tenant控制台正常，但通过nginx-ingress暴露到公网时，所有客户端请求均返回HTTP 400错误
• API和控制台的独立Ingress定义均出现相同问题，上游目标为StatefulSet中Pod的IP，但Pod无相关日志输出

Nginx日志信息 #

2024/01/11 16:38:55 [error] 2469#2469: *16521012 recv() failed (104: Connection reset by peer) while reading upstream, client: XXX.XXX.XXX.XXX, server: XXX, request: "GET / HTTP/2.0", upstream: "http://172.28.1.49:9443/", host: "XXX"

Curl命令测试输出 #

user ~ $ curl GET --verbose https://xxx.xxx/login
* Could not resolve host: GET
* Closing connection 0
curl: (6) Could not resolve host: GET
*   Trying LB-IP:443...
* Connected to xxx.xxx (LB-IP) port 443 (#1)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=xxx.xxx
*  start date: Jan  9 16:32:45 2024 GMT
*  expire date: Apr  8 16:32:44 2024 GMT
*  subjectAltName: host "xxx.xxx" matched cert's "xxx.xxx"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* h2 [:method: GET]
* h2 [:scheme: https]
* h2 [:authority: xxx.xxx]
* h2 [:path: /login]
* h2 [user-agent: curl/8.1.2]
* h2 [accept: */*]
* Using Stream ID: 1 (easy handle 0x133008200)
> GET /login HTTP/2
> Host: xxx.xxx
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/2 400
< date: Fri, 12 Jan 2024 08:31:20 GMT
< strict-transport-security: max-age=15724800; includeSubDomains
<
Client sent an HTTP request to an HTTPS server.
* Connection #1 to host xxx.xxx left intact

当前ArgoCD应用配置 #

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: XXX
  namespace: argocd
  annotations:
    # ...
spec:
  destination:
    namespace: minio-tenant
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://operator.min.io
    chart: tenant
    targetRevision: "5.0.11"
    helm:
      version: v3
      valuesObject:
        tenant:
          name: XXX
          pools:
            - servers: 4
          exposeServices:
            minio: false
            console: false
        ingress:
          api:
            enabled: true
            ingressClassName: "nginx"
            annotations:
              nginx.ingress.kubernetes.io/ssl-redirect: "true"
              nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
              cert-manager.io/cluster-issuer: "letsencrypt-prod"
            tls:
            - hosts:
              - XXXXXX
              secretName: XXX
            host: XXXXXX
            path: /
            pathType: Prefix
          console:
            enabled: true
            ingressClassName: "nginx"
            annotations:
              nginx.ingress.kubernetes.io/ssl-redirect: "true"
              nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
              cert-manager.io/cluster-issuer: "letsencrypt-prod"
            tls:
            - hosts:
              - XXXXXX
              secretName: XXX
            host: XXXXXX
            path: /
            pathType: Prefix

已尝试的Nginx注解配置 #

nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_set_header Host $http_host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_set_header X-NginX-Proxy true;
  real_ip_header X-Real-IP;
  chunked_transfer_encoding off;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";
nginx.ingress.kubernetes.io/proxy-http-version: "1.1"

问题根源 #

从Nginx日志的upstream: "http://172.28.1.49:9443/"可以看出，Nginx正在使用HTTP协议转发请求到MinIO Pod的9443端口，但MinIO的9443端口默认是HTTPS服务端口，协议不匹配导致返回"Client sent an HTTP request to an HTTPS server"错误。

解决步骤 #

1. 添加HTTPS上游转发注解
   在Ingress的annotations中新增配置，强制Nginx使用HTTPS协议连接上游MinIO服务：

nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

2. 更新ArgoCD Helm配置
   修改ArgoCD应用的Helm valuesObject，为API和Console的Ingress分别添加上述注解：

ingress:
  api:
    annotations:
      # 保留原有注解
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  console:
    annotations:
      # 保留原有注解
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

3. 验证修复效果
   更新ArgoCD应用后，重新执行curl测试：

curl --verbose https://xxx.xxx/login

此时Nginx会用HTTPS协议转发请求到MinIO的9443端口，协议匹配后即可正常访问。

4. 额外检查项
   如果问题仍存在，确认：

• MinIO Tenant的服务是否正确暴露了HTTPS端口
• Pod内MinIO配置是否由Operator自动启用了HTTPS（默认会生成自签名证书）

内容的提问来源于stack exchange，提问作者tech4242