Kubernetes多Ingress Controller抢占Ingress地址问题求助

基础设施背景 #

我的Kubernetes（K3s）集群包含4个节点：

k3s-server location=home   (VM) 主要用于etcd复制
k3s-agent  location=home   (VM) 运行大多数Pod
mercury    location=home   (RPI4) 重要Pod的备份节点
moon       location=cloud  (Cloud VM) 在公有云中运行特定工作负载

我在两个命名空间中部署了两个不同的ingress-nginx Helm实例：

• 家庭本地服务用：ingressClassName=nginx，loadBalancerIP: 192.168.113.230，运行在nginx-ingres-home命名空间
• 云环境用：ingressClassName=nginx-cloud，loadBalancerIP: 91.x.x.x，运行在nginx-ingress-cloud命名空间

问题现象 #

定义了多个使用不同IngressClass的Ingress资源，但执行kubectl get ingress -A发现所有Ingress的ADDRESS字段始终显示其中一个控制器的地址，且会周期性切换：

NAMESPACE              NAME                   CLASS         HOSTS                        ADDRESS           PORTS     AGE
kubernetes-dashboard   kubernetes-dashboard   nginx         k3s.local.example.com        192.168.113.230   80, 443   4d3h
longhorn-system        longhorn-ingress       nginx         longhorn.local.example.com   192.168.113.230   80, 443   10d
mailu                  mailu                  nginx-cloud   mail.example.com             192.168.113.230   80, 443   2d23h
pihole                 pihole                 nginx         dns.local.example.com        192.168.113.230   80, 443   10d
ubiquiti               unifi-web-interface     nginx         unifi.local.example.com       192.168.113.230   80, 443   24h

nginx-ingress-home日志显示控制器每隔1分钟就会更新所有Ingress的地址：

I0510 19:46:12.087815       7 status.go:300] "updating Ingress status" namespace="pihole" ingress="pihole" currentValue=[{IP:91.x.x.x Hostname: Ports:[]}] newValue=[{IP:192.168.113.230 Hostname: Ports:[]}]
I0510 19:46:12.087857       7 status.go:300] "updating Ingress status" namespace="ubiquiti" ingress="unifi-web-interface" currentValue=[{IP:91.x.x.x Hostname: Ports:[]}] newValue=[{IP:192.168.113.230 Hostname: Ports:[]}]
I0510 19:46:12.088485       7 status.go:300] "updating Ingress status" namespace="mailu" ingress="mailu" currentValue=[{IP:91.x.x.x Hostname: Ports:[]}] newValue=[{IP:192.168.113.230 Hostname: Ports:[]}]
I0510 19:46:12.088782       7 status.go:300] "updating Ingress status" namespace="longhorn-system" ingress="longhorn-ingress" currentValue=[{IP:91.x.x.x Hostname: Ports:[]}] newValue=[{IP:192.168.113.230 Hostname: Ports:[]}]
I0510 19:46:12.090051       7 status.go:300] "updating Ingress status" namespace="kubernetes-dashboard" ingress="kubernetes-dashboard" currentValue=[{IP:91.x.x.x Hostname: Ports:[]}] newValue=[{IP:192.168.113.230 Hostname: Ports:[]}]

nginx-ingress-cloud也会执行相同操作，将地址替换为91.x.x.x。需要让每个控制器只更新匹配自身IngressClass的Ingress资源。

配置信息 #

nginx-ingress-home的Helm配置值 #

controller:
  ingressClass: "nginx"
  ingressClassResource:
    name: nginx
    enabled: yes
    default: yes
  service:
    type: "LoadBalancer"
    loadBalancerIP: 192.168.113.230
  nodeSelector:
    location: home
  tolerations: #允许在备份节点上运行
    - key: "backup"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
  affinity: #优先调度到标记为type=power的节点
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          preference:
            matchExpressions:
              - key: type
                operator: In
                values:
                  - power

nginx-ingress-cloud的Helm配置值 #

controller:
  ingressClass: "nginx-cloud"
  ingressClassResource:
    name: nginx-cloud
    enabled: yes
    default: no
  service:
    type: "LoadBalancer"
    loadBalancerIP: 91.x.x.x
  nodeSelector:
    location: cloud

家庭环境Ingress示例（Kubernetes Dashboard） #

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - k3s.local.example.com
      secretName: kubernetes-dashboard-tls
  rules:
    - host: k3s.local.example.com
      http:
        paths:
          - backend:
              service:
                name: kubernetes-dashboard
                port:
                  number: 443

            path: /
            pathType: Prefix

解决方案 #

问题出在两个Ingress Controller默认会处理所有Ingress资源，哪怕这些资源的IngressClass不匹配。需要给每个控制器添加IngressClass过滤规则，让它们只关注对应IngressClass的资源。

1. 更新Helm配置，添加watchIngressClassOnly参数 #

在每个Ingress Controller的Helm values中，加入controller.watchIngressClassOnly: true，这个参数会让控制器只处理与自身配置的ingressClass匹配的Ingress资源。

更新后的nginx-ingress-home配置

controller:
  ingressClass: "nginx"
  ingressClassResource:
    name: nginx
    enabled: yes
    default: yes
  watchIngressClassOnly: true # 新增过滤规则
  # 其他原有配置保持不变

更新后的nginx-ingress-cloud配置

controller:
  ingressClass: "nginx-cloud"
  ingressClassResource:
    name: nginx-cloud
    enabled: yes
    default: no
  watchIngressClassOnly: true # 新增过滤规则
  # 其他原有配置保持不变

2. 重新部署Helm实例 #

执行Helm upgrade命令更新两个控制器：

# 更新家庭环境控制器
helm upgrade nginx-ingress-home ingress-nginx/ingress-nginx -n nginx-ingres-home -f home-values.yaml

# 更新云环境控制器
helm upgrade nginx-ingress-cloud ingress-nginx/ingress-nginx -n nginx-ingress-cloud -f cloud-values.yaml

3. 验证效果 #

等待控制器重启后，执行kubectl get ingress -A，每个Ingress的ADDRESS应该会显示对应IngressClass控制器的IP，且不会再出现周期性切换的情况。同时查看控制器日志，应该只会看到匹配自身IngressClass的Ingress状态更新。

内容的提问来源于stack exchange，提问作者Florian7843