AKS集群Ingress控制器返回403-Forbidden错误排查求助

已完成以下配置步骤：

• 为abc.com创建Azure DNS zone
• 在GoDaddy的DNS设置中更新Azure DNS zone生成的域名服务器
• 为K8s集群agentpool添加'DNS zone contributor'角色
• 部署Ingress controller、External DNS、Cert-manager、ClusterIssuer
• 配置应用的Ingress路由
• 在集群中部署应用，创建对应的Pod和Service

错误现象 #

通过Postman访问API https://dev.abc.com/backend/usermanager/api/user 时收到403-Forbidden错误。在集群内的Ingress controller容器中执行curl请求该API，同样返回403-Forbidden，日志如下：

bash-5.1$ curl -v -d '{"name": "John Doe", "mobileNumber": "1234554321"}' -H 'Content-Type: 
application/json' https://dev.abc.com/backend/usermanager/api/user
*   Trying 21.81.66.166:443...
* Connected to dev.abc.com (21.81.66.166) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=dev.abc.com
*  start date: Jul 20 03:29:46 2022 GMT
*  expire date: Oct 18 03:29:45 2022 GMT
*  subjectAltName: host "dev.abc.com" matched cert's "dev.abc.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: POST]
* h2h3 [:path: /backend/usermanager/api/user]
* h2h3 [:scheme: https]
* h2h3 [:authority: dev.abc.com]
* h2h3 [user-agent: curl/7.83.1]
* h2h3 [accept: */*]
* h2h3 [content-type: application/json]
* h2h3 [content-length: 50]
* Using Stream ID: 1 (easy handle 0x64161d71800)
> POST /backend/usermanager/api/user HTTP/2
> Host: dev.abc.com
> user-agent: curl/7.83.1
> accept: */*
> content-type: application/json
> content-length: 50
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* We are completely uploaded and fine
< HTTP/2 403
< date: Fri, 22 Jul 2022 04:40:59 GMT
< content-length: 0
< vary: Origin
< vary: Access-Control-Request-Method
< vary: Access-Control-Request-Headers
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: 0
< strict-transport-security: max-age=15724800; includeSubDomains
< x-frame-options: DENY
<
* Connection #0 to host dev.abc.com left intact
bash-5.1$ * We are completely uploaded and fine
< HTTP/2 403
< date: Fri, 22 Jul 2022 04:40:59 GMT
< content-length: 0
< vary: Origin
< vary: Access-Control-Request-Method
< vary: Access-Control-Request-Headers
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: 0
< strict-transport-security: max-age=15724800; includeSubDomains
< x-frame-options: DENY

正常访问情况 #

在Ingress controller容器中直接通过Service名称访问该端点时请求正常，日志如下：

bash-5.1$ curl -v -d '{"name": "John Doe", "mobileNumber": "1234554321"}' -H 'Content-Type: application/json' http://usermanager:8082/api/user
*   Trying 10.0.236.14:8082...
* Connected to usermanager (10.0.236.14) port 8082 (#0)
> POST /api/user HTTP/1.1
> Host: usermanager:8082
> User-Agent: curl/7.83.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 50
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< Vary: Origin
< Vary: Access-Control-Request-Method
< Vary: Access-Control-Request-Headers
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Content-Type: application/json
< Transfer-Encoding: chunked
< Date: Fri, 22 Jul 2022 03:58:33 GMT

Ingress路由配置 #

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: abc-ingress
  namespace: k8sns-abc-dev
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
  - hosts:
    - dev.abc.com
    secretName: tls-secret
  rules:
    - host: dev.abc.com
      http:
        paths:
          - path: /backend/usermanager
            pathType: Prefix
            backend:
              service:
                name: usermanager
                port:
                  number: 8082

请求协助排查并修复通过DNS主机访问API时出现403-Forbidden错误的问题。

内容的提问来源于stack exchange，提问作者Dhaval Simaria