Kubernetes集群中Nginx Ingress配置及外部访问问题咨询

背景 #

我刚接触Kubernetes，在虚拟机上搭建本地集群并让至少我的宿主机能够访问它时遇到了一些小问题。

集群配置情况 #

基础环境 #

• 本地集群由3台Debian虚拟机组成：1个master节点，2个worker节点。我创建了一个CIDR为10.0.0.0/16的虚拟接口，DHCP域设置为abcd.com。
• 创建集群时，我设置了clusterDNS: abcd.com、podCIDR: 69.96.0.0/16和serviceCIDR: 69.97.0.0/16，使用Cilium作为CNI，CoreDNS作为DNS服务，Corefile配置如下：

.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes abcd.com in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30
    loop
    reload
    loadbalance
}

• 在default命名空间部署了nginx-pod，并创建了一个NodePort类型的服务nginx-svc，将nginx-pod的80端口暴露出来。同时创建了Ingress规则，将abcd.com的请求转发到nginx-svc。
• 已安装ingress-nginx-controller。

相关日志信息 #

Ingress资源详情 #

ayush@ip-10-0-191-144:~$ kubectl describe ingress/example
Name:             example
Labels:           <none>
Namespace:        default
Address:
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host           Path  Backends
  ----           ----  --------
  abcd.com
                 /   nginx-svc:80 (69.96.1.111:80)
Annotations:     <none>
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    30m (x5 over 11h)  nginx-ingress-controller  Scheduled for sync

集群服务列表 #

ayush@ip-10-0-191-144:~$ kubectl get svc -A
NAMESPACE       NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
default         kubernetes                           ClusterIP      69.97.0.1       <none>        443/TCP                      2d14h
default         nginx-svc                            NodePort       69.97.216.124   <none>        80:30744/TCP                 26h
ingress-nginx   ingress-nginx-controller             LoadBalancer   69.97.88.241    <pending>     80:31565/TCP,443:32581/TCP   2d2h
ingress-nginx   ingress-nginx-controller-admission   ClusterIP      69.97.200.46    <none>        443/TCP                      2d2h
kube-system     cilium-agent                         ClusterIP      None            <none>        9964/TCP                     2d14h
kube-system     hubble-metrics                       ClusterIP      None            <none>        9965/TCP                     2d14h
kube-system     hubble-peer                          ClusterIP      69.97.190.83    <none>        443/TCP                      2d14h
kube-system     hubble-relay                         ClusterIP      69.97.188.92    <none>        80/TCP                       2d14h
kube-system     hubble-ui                            ClusterIP      69.97.9.207     <none>        80/TCP                       2d14h
kube-system     kube-dns                             ClusterIP      69.97.0.10      <none>        53/UDP,53/TCP,9153/TCP       2d14h

集群Pod列表 #

ayush@ip-10-0-191-144:~$ kubectl get pods -A -o wide
NAMESPACE       NAME                                        READY   STATUS    RESTARTS       AGE     IP             NODE              NOMINATED NODE   READINESS GATES
default         dnsutils                                    1/1     Running   1 (11h ago)    11h     69.96.2.47     ip-10-0-94-54     <none>           <none>
default         nginx-pod                                   1/1     Running   4 (11h ago)    26h     69.96.1.111    ip-10-0-38-41     <none>           <none>
ingress-nginx   ingress-nginx-controller-55474d95c5-f4kfx   1/1     Running   4 (11h ago)    2d3h    69.96.1.80     ip-10-0-38-41     <none>           <none>
kube-system     cilium-428bt                                1/1     Running   7 (11h ago)    2d14h   10.0.38.41     ip-10-0-38-41     <none>           <none>
kube-system     cilium-8hpbb                                1/1     Running   4 (11h ago)    2d14h   10.0.191.144   ip-10-0-191-144   <none>           <none>
kube-system     cilium-cxgnc                                1/1     Running   6 (11h ago)    2d14h   10.0.94.54     ip-10-0-94-54     <none>           <none>
kube-system     cilium-operator-6cdc4568cb-gg58q            1/1     Running   39 (11h ago)   2d14h   10.0.38.41     ip-10-0-38-41     <none>           <none>
kube-system     cilium-operator-6cdc4568cb-k7nsd            1/1     Running   4 (11h ago)    2d14h   10.0.191.144   ip-10-0-191-144   <none>           <none>
kube-system     coredns-76f75df574-45f7b                    1/1     Running   9 (11h ago)    2d14h   69.96.0.43     ip-10-0-191-144   <none>           <none>
kube-system     coredns-76f75df574-77m4h                    1/1     Running   9 (11h ago)    2d14h   69.96.0.221    ip-10-0-191-144   <none>           <none>
kube-system     etcd-ip-10-0-191-144                        1/1     Running   11 (11h ago)   2d14h   10.0.191.144   ip-10-0-191-144   <none>           <none>
kube-system     hubble-relay-d8b6b55c9-st57x                1/1     Running   9 (11h ago)    2d11h   69.96.2.197    ip-10-0-94-54     <none>           <none>
kube-system     hubble-ui-6548d56557-b6f49                  2/2     Running   9 (11h ago)    2d11h   69.96.2.93     ip-10-0-94-54     <none>           <none>
kube-system     kube-apiserver-ip-10-0-191-144              1/1     Running   11 (11h ago)   2d14h   10.0.191.144   ip-10-0-191-144   <none>           <none>
kube-system     kube-controller-manager-ip-10-0-191-144     1/1     Running   11 (11h ago)   2d14h   10.0.191.144   ip-10-0-191-144   <none>           <none>
kube-system     kube-proxy-7gc2j                            1/1     Running   4 (11h ago)    2d14h   10.0.94.54     ip-10-0-94-54     <none>           <none>
kube-system     kube-proxy-xt8nc                            1/1     Running   4 (11h ago)    2d14h   10.0.191.144   ip-10-0-191-144   <none>           <none>
kube-system     kube-proxy-xzjjj                            1/1     Running   4 (11h ago)    2d14h   10.0.38.41     ip-10-0-38-41     <none>           <none>
kube-system     kube-scheduler-ip-10-0-191-144              1/1     Running   11 (11h ago)   2d14h   10.0.191.144   ip-10-0-191-144   <none>           <none>

问题咨询 #

1. 虽然nginx-ingress-controller显示为Running状态，但它一直无法获取外部IP，停留在<pending>状态（可以在上面的服务列表中看到），我不确定原因是什么！

2. 通过任意节点的IP，我可以通过curl -H 'Host: isitayush.dev' -X GET 10.0.191.144:31565访问nginx-svc，但我想实现直接通过节点的外部IP就能访问到nginx-svc或nginx-pod，该怎么做？

我能不能把nginx-ingress-controller的端口从31565改成80或443，这样我就能用节点的外部IP直接访问到规则中定义的服务？

3. 是否可以通过域名abcd.com或者其他域名（比如我拥有的pqrs.com）从外部访问nginx-pod？我已经给pqrs.com设置了A记录，指向外部负载均衡器IP或者nginx-ingress-controller所在节点的IP。

我的最终目标是设置A记录，将我拥有的多个域名指向这个集群（使用一个外部负载均衡器在所有节点IP或Pod IP之间做负载均衡），然后通过nginx-ingress-controller根据Ingress规则将请求路由到对应的服务。这样可行吗？我打算把集群部署在AWS上，尽量不想使用他们的负载均衡器服务。

4. 能不能把DHCP域和集群DNS域分开？比如在接口设置中保留abcd.com，创建集群时设置pqrs.com作为集群DNS域？

感谢帮助！

备注：内容来源于stack exchange，提问作者Ayush