WSL2+Docker Desktop环境下K8s Ingress的HTTPS服务无法被同局域网Linux机器访问的问题排查求助

我在Windows 11 Pro的Docker Desktop + WSL2环境里的Kubernetes集群部署了一个认证服务，从Windows本机（WSL2外部）用CURL或PostMan能正常访问它的接口，但同局域网内的Linux机器却没法通过Ingress的HTTPS访问这个Rest接口。

为了验证服务本身的连通性没问题，我创建了一个NodePort类型的服务，通过kubectl get svc拿到外部端口后，不管是Windows还是Linux机器都能通过HTTP正常请求并得到正确响应，但这个方式完全绕开了K8s Ingress的SSL/TLS（Ingress用的是自签名假证书），只能走HTTP，没法满足我的需求——因为我需要从认证服务的响应里获取Cookie，所以HTTPS是必不可少的，但走Ingress的HTTPS请求完全得不到响应。

我已经在Windows机器上做了这些尝试：

• 开放了80、443（甚至22）端口
• 把Linux机器加到了Windows防火墙的可信计算机列表里
• 临时关闭过防火墙测试

但请求还是被无声丢弃，我怀疑是Ingress的问题，但缺乏分析Ingress内部情况的工具，所以先把我的环境配置和现象整理出来，寻求大家的帮助。

一、K8s相关配置文件 #

认证服务的Deployment和ClusterIP Service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gts-auth-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gts-auth
  template:
    metadata:
      labels:
        app: gts-auth
    spec:
      containers:
      - name: gts-auth
        image: creativewarlock/gts-auth
        imagePullPolicy: IfNotPresent
        env:
        - name: MONGO_URI
          value: 'mongodb://gts-auth-mongo-srv:27017/auth'
        - name: JWT_KEY
          valueFrom:
            secretKeyRef:
              name: gts-jwt-secret
              key: JWT_KEY
      imagePullSecrets:
      - name: docker-hub
---
apiVersion: v1
kind: Service
metadata:
  name: gts-auth-srv
spec:
  type: ClusterIP
  selector:
    app: gts-auth
  ports:
  - name: gts-auth
    protocol: TCP
    port: 3000
    targetPort: 3000

Ingress配置（包含认证服务和Web客户端路由）

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gts-ingress-srv
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/use-regex: 'true'
spec:
  rules:
  - host: gts.dev # Windows本机hosts文件已配置：127.0.0.1 gts.dev（也试过localhost）
    http:
      paths:
      - path: /api/users/?(.*)    # 匹配/api/users/signin等路径
        pathType: Prefix
        backend:
          service:
            name: gts-auth-srv
            port:
              number: 3000
      - path: /?(.*)                # 其他请求转发到Web客户端
        pathType: Prefix
        backend:
          service:
            name: gts-client-srv
            port:
              number: 8080

用于外部直接访问的NodePort Service

apiVersion: v1
kind: Service
metadata:
  name: gts-auth-srv-ext
spec:
  type: NodePort
  selector:
    app: gts-auth
  ports:
  - name: gts-auth-srv
    protocol: TCP
    port: 3000
    targetPort: 3000

二、Ingress控制器Pod的描述信息 #

kubectl describe pod ingress-nginx-controller-6bf7bc7f94-krv2c -n ingress-nginx

Name:         ingress-nginx-controller-6bf7bc7f94-krv2c
Namespace:    ingress-nginx
Priority:     0
Node:         docker-desktop/192.168.65.4
Start Time:   Thu, 11 Aug 2022 12:31:42 +0200
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/name=ingress-nginx
              pod-template-hash=6bf7bc7f94
Annotations:  <none>
Status:       Running
IP:           10.1.28.235
IPs:
  IP:           10.1.28.235
Controlled By:  ReplicaSet/ingress-nginx-controller-6bf7bc7f94
Containers:
  controller:
    Container ID:  docker://0cef9bec68d309011a32e18bf4f3a2a7ec7050b3db6cc7bafd87565e370f6695
    Image:         registry.k8s.io/ingress-nginx/controller:v1.3.0@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5
    Image ID:      docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-controller-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Sun, 19 Feb 2023 11:50:07 +0100
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Sat, 18 Feb 2023 11:44:18 +0100
      Finished:     Sun, 19 Feb 2023 11:49:36 +0100
    Ready:          True
    Restart Count:  132
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-6bf7bc7f94-krv2c (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hpxhw (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-hpxhw:
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  RELOAD  54m (x3 over 95m)  nginx-ingress-controller  NGINX reload triggered due to a change in configuration

三、Ingress控制器日志 #

kubectl logs -f ingress-nginx-controller-6bf7bc7f94-krv2c -n ingress-nginx

W0219 10:50:08.009613       7 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0219 10:50:08.010796       7 main.go:230] "Creating API client" host="https://10.96.0.1:443"
W0219 10:50:30.162337       7 main.go:271] Initial connection to the Kubernetes API server was retried 1 times.
I0219 10:50:30.162386       7 main.go:274] "Running in Kubernetes cluster" major="1" minor="24" git="v1.24.1" state="clean" commit="3ddd0f45aa91e2f30c70734b175631bec5b5825a" platform="linux/amd64"
I0219 10:50:30.278395       7 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0219 10:50:30.299891       7 ssl.go:531] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0219 10:50:30.314983       7 nginx.go:258] "Starting NGINX Ingress controller"
I0219 10:50:30.321843       7 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"407f816e-98b9-4ab6-a08f-95efafa21ddc", APIVersion:"v1", ResourceVersion:"612", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0219 10:50:31.516611       7 nginx.go:301] "Starting NGINX process"
I0219 10:50:31.516698       7 leaderelection.go:248] attempting to acquire leader lease ingress-nginx/ingress-controller-leader...
I0219 10:50:31.517522       7 nginx.go:321] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0219 10:50:31.517893       7 controller.go:167] "Configuration changes detected, backend reload required"
I0219 10:50:31.543621       7 leaderelection.go:258] successfully acquired lease ingress-nginx/ingress-controller-leader
I0219 10:50:31.543648       7 status.go:84] "New leader elected" identity="ingress-nginx-controller-6bf7bc7f94-krv2c"
I0219 10:50:31.610775       7 controller.go:184] "Backend successfully reloaded"
I0219 10:50:31.610910       7 controller.go:195] "Initial sync, sleeping for 1 second"
192.168.65.4 - - [19/Feb/2023:13:09:46 +0000] "POST /api/users/signin/ HTTP/1.1" 200 150 "-" "curl/7.83.1" 259 0.054 [default-gts-auth-srv-3000] [] 10.1.28.246:3000 150 0.050 200 5b8761d90f76b4c160b0a2effc980b85

注：日志里只有Windows本机请求的记录，Linux机器的HTTPS请求完全没出现在日志里，但Linux机器收到了NGINX的404响应。

四、Linux机器上的请求测试 #

执行的curl命令：

Blacky@Kaspar:~$ curl -k -H 'Content-Type: application/json' https://192.168.0.57/api/users/signin/ -X POST -d '{ "email": "DummyUser@dummies.com", "password": "iHhRCF9Dm3wKT" }'

返回的结果：

<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

五、Windows上的端口转发和防火墙配置脚本 #

我用PowerShell脚本配置了端口转发和防火墙规则：

$wsl_ip = (ubuntu.exe -c "ifconfig eth0 | grep 'inet '").trim().split()| where {$_}
$regex = [regex] "\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b"
$ip_array = $regex.Matches($wsl_ip) | %{ $_.value }
$wsl_ip = $ip_array[0]

Write-Host "WSL Machine IP: ""$wsl_ip"""

New-NetFireWallRule -DisplayName 'WSL SSL' -Direction Outbound -LocalPort 443 -Action Allow -Protocol TCP
New-NetFireWallRule -DisplayName 'WSL SSL' -Direction Inbound -LocalPort 443 -Action Allow -Protocol TCP

#### ------------ Clear all portproxy rules ------------ ####
netsh int portproxy reset all

#### ------------ SSH rule - port 22 ------------ ####
netsh interface portproxy add v4tov4 listenport=22 listenaddress=0.0.0.0 connectport=22 connectaddress=$wsl_ip

#### ------------ Webserver SSL rule - port 443 ------------ ####
netsh interface portproxy add v4tov4 listenport=443 listenaddress=0.0.0.0 connectport=443 connectaddress=$wsl_ip

#### ------------ Webserver SSL rule - port 80 ------------ ####
netsh interface portproxy add v4tov4 listenport=80 listenaddress=0.0.0.0 connectport=80 connectaddress=$wsl_ip

我的疑问和求助点 #

• 是不是K8s的自签名假证书导致了这个问题？
• 我需要为局域网内的使用申请Let's Encrypt证书吗？
• 除了kubectl logs -f <ingress-pod> -n ingress-nginx，还有什么工具可以分析Ingress内部