Kubernetes裸金属集群cert-manager HTTP-01挑战传播检查失败（404/连接拒绝）问题求助

我正在尝试给本地裸金属Kubernetes集群配置HTTPS，但遇到了cert-manager的错误：

cert-manager/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'"

我怀疑是NodePort的配置有问题，但不确定怎么修复。目前能想到的临时办法是添加http://<domain>.com/.well-known/acme-challenge/作为路由，但这显然不是长期解决方案。

我的配置文件 #

ingress-service.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/use-regex: "true"
    # nginx.ingress.kubernetes.io/rewrite-target: /$1
    cert-manager.io/cluster-issuer: letsencrypt-prod # tell ingress to use https
    # nginx.ingress.kubernetes.io/ssl-redirect: 'true' # redirect from http to https
spec:
  tls:
  - hosts:
    - <domain>.com
    - www.<domain>.com
    secretName: secret-redacted-com
  rules:
  - host: <domain>.com
    http:
      paths:
      - path: /?(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: server-cluster-ip-service
            port:
              number: 8888
  - host: www.<domain>.com
    http:
      paths:
      - path: /?(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: server-cluster-ip-service
            port:
              number: 8888

cluster-issuer.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: redacted@gmail.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-redacted-prod
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          # class: nginx
          ingressTemplate:
            metadata:
              annotations:
                kubernetes.io/ingress.class: nginx

server-cluster-ip-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: server-cluster-ip-service
spec:
  type: ClusterIP
  selector:
    component: server
  ports:
  - port: 8888
    targetPort: 8888

server-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: server-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      component: server
  template:
    metadata:
      labels:
        component: server
    spec:
      containers:
      - name: server
        image: spoonobi/multi-server-arm
        ports:
        - containerPort: 8888

service-nodeports.yaml

apiVersion: v1
kind: Service
metadata:
  name: server-nodeports
spec:
  type: NodePort
  selector:
    component: server
  ports:
  - name: http
    port: 80
    targetPort: 8888
    nodePort: 30602
  - name: https
    port: 443
    targetPort: 8888
    nodePort: 30824

相关日志 #

cert-manager pod日志 (kubectl logs cert-manager-7fb78674d7-8l8v4 -n cert-manager)

I0109 14:21:46.934907       1 start.go:75] cert-manager "msg"="starting controller" "git-commit"="a96bae172ddb1fcd4b57f1859ab9d1a9e94f7451" "version"="v1.10.1"
I0109 14:21:46.935260       1 controller.go:242] cert-manager/controller/build-context "msg"="configured acme dns01 nameservers" "nameservers"=["10.96.0.10:53"]
W0109 14:21:46.942385       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0109 14:21:47.020721       1 controller.go:70] cert-manager/controller "msg"="enabled controllers: [certificaterequests-approver certificaterequests-issuer-acme certificaterequests-issuer-ca certificaterequests-issuer-selfsigned certificaterequests-issuer-vault certificaterequests-issuer-venafi certificates-issuing certificates-key-manager certificates-metrics certificates-readiness certificates-request-manager certificates-revision-manager certificates-trigger challenges clusterissuers ingress-shim issuers orders]"
I0109 14:21:47.026062       1 controller.go:134] cert-manager/controller "msg"="starting leader election"
I0109 14:21:47.028151       1 leaderelection.go:248] attempting to acquire leader lease kube-system/cert-manager-controller...
I0109 14:21:47.029212       1 controller.go:91] cert-manager/controller "msg"="starting metrics server" "address"={"IP":"::","Port":9402,"Zone":""}
I0109 14:21:47.164717       1 leaderelection.go:258] successfully acquired lease kube-system/cert-manager-controller
I0109 14:21:47.189500       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-ca"
I0109 14:21:47.190713       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-venafi"
I0109 14:21:47.192837       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="gateway-shim"
I0109 14:21:47.193391       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="orders"
I0109 14:21:47.193503       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-request-manager"
I0109 14:21:47.205713       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-approver"
I0109 14:21:47.208892       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-venafi"
I0109 14:21:47.213435       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-ca"
I0109 14:21:47.213533       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-selfsigned"
I0109 14:21:47.216346       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-readiness"
I0109 14:21:47.218932       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="ingress-shim"
I0109 14:21:47.221861       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-acme"
I0109 14:21:47.234795       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-revision-manager"
I0109 14:21:47.241310       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-acme"
I0109 14:21:47.242004       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-trigger"
I0109 14:21:47.256862       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="issuers"
I0109 14:21:47.257295       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-selfsigned"
I0109 14:21:47.257529       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-issuing"
I0109 14:21:47.266041       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-key-manager"
I0109 14:21:47.267554       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-vault"
I0109 14:21:47.267654       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-metrics"
I0109 14:21:47.269328       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="clusterissuers"
I0109 14:21:47.385919       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="challenges"
I0109 14:21:47.393223       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-vault"
I0109 14:21:47.470499       1 setup.go:202] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-secret-prod" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-prod" "resource_namespace"="" "resource_version"="v1"
I0109 14:21:47.490596       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="ecmatrials.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-2rzhm" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="secret-ecmatrials-com-pxg88-96326727-78145582" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0109 14:21:47.490592       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="www.ecmatrials.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-prw2b" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="secret-ecmatrials-com-pxg88-96326727-665678090" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0109 14:21:47.491386       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="www.ecmatrials.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-ctsfv" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="secret-ecmatrials-com-pxg88-96326727-665678090" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0109 14:21:47.491727       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="www.ecmatrials.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-mhrwj" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="secret-ecmatrials-com-pxg88-96326727-665678090" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0109 14:21:47.491392       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="ecmatrials.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-q29l7" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="secret-ecmatrials-com-pxg88-96326727-78145582" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0109 14:21:47.493451       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="ecmatrials.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-chbv4" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="secret-ecmatrials-com-