将WebSocket连接从domain-1转发至domain-2时出现400错误求助

背景需求 #

需要将WebSocket(WS)连接从domain-1转发至domain-2：

• domain-1使用弱加密算法、TLS 1.2及短URL对外提供服务
• domain-2仅支持TLS 1.3强加密算法，且域名较长

当前转发流程中出现400错误，具体信息如下：

错误日志 #

domain-1的NGINX日志 #

2023/10/06 13:17:01 [notice] 1152#1152: *37717323 "(?i)/std(/|$)(.*)" matches "/std/base/key1/key2/name1", client: 10.0.179.122, server: domain-1, request: "GET /std/base/key1/key2/name1 HTTP/1.1", host: "domain-1"

2023/10/06 13:17:01 [notice] 1152#1152: *37717323 rewritten data: "/base/key1/key2/name1", args: "", client: 10.0.179.122, server: domain-1, request: "GET /std/base/key1/key2/name1 HTTP/1.1", host: "domain-1"

10.0.179.122 - - [06/Oct/2023:13:17:01 +0000] "GET /std/base/key1/key2/name1 HTTP/1.1" 400 248 "-" "-" 336 0.004 [standard-external-service-443] [] 3.79.135.121:443 248 0.005 400 3111ad9748961e237011dd0449dafe08

domain-2的NGINX日志 #

2023/10/10 13:17:01 [info] 140#140: *40730708 client sent plain HTTP request to HTTPS port while reading client request headers, client: 10.0.62.0, server: _, request: "GET /base/key1/key2/CS-SIEMENS HTTP/1.1", host: "domain-1"

当前环境配置 #

运行在K8s环境，使用NGINX Ingress + External Service，相关配置如下：

External Service配置 #

apiVersion: v1
kind: Service
metadata:
  name: standard-external-service
  namespace: qa
spec:
  type: ExternalName
  externalName: domain-2

Ingress配置 #

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-standard-external
  namespace: qa
  annotations:
    cert-manager.io/issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/enable-access-log: "true"
    nginx.ingress.kubernetes.io/enable-rewrite-log: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 1m
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/websocket-services: standard-external-service
    nginx.org/websocket-services: standard-external-service
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - "domain-1"
        - "*.domain-1"
      secretName: tls-secret-qa
  rules:
    - host: "domain-1"
      http:
        paths:
          # e-Mobility Standard
          - path: /std(/|$)(.*)
            pathType: Prefix
            backend:
              service:
                name: standard-external-service
                port:
                  number: 443
    - host: "*.domain-1"
      http:
        paths:
          # e-Mobility Standard
          - path: /std(/|$)(.*)
            pathType: Prefix
            backend:
              service:
                name: standard-external-service
                port:
                  number: 443

NGINX Controller ConfigMap配置 #

{
  "apiVersion": "v1",
  "data": {
    "allow-snippet-annotations": "true",
    "keep-alive": "349",
    "large-client-header-buffers": "4 16k",
    "ssl-ciphers": "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:AES256-SHA256",
    "ssl-protocols": "TLSv1.3 TLSv1.2"
  },
  "kind": "ConfigMap",
  "metadata": {
    "labels": {
      "app.kubernetes.io/component": "controller",
      "app.kubernetes.io/instance": "ingress-nginx",
      "app.kubernetes.io/managed-by": "Helm",
      "app.kubernetes.io/name": "ingress-nginx",
      "app.kubernetes.io/part-of": "ingress-nginx",
      "app.kubernetes.io/version": "1.8.1",
      "argocd.argoproj.io/instance": "ingress-nginx",
      "helm.sh/chart": "ingress-nginx-4.7.1"
    },
    "name": "ingress-nginx-controller",
    "namespace": "ingress-nginx"
  }
}

问题疑问 #

已确认NGINX完成URL重写并转发请求，但出现*Client sent plain HTTP request to HTTPS port*的400错误。原本预期SSL在NGINX层终止后，以HTTPS方式转发至domain-2，不清楚为何会出现明文HTTP发送到HTTPS端口的错误，需要协助排查原因。

内容的提问来源于stack exchange，提问作者LucasBrazi06