暴露内部Service时Ingress使用无效命名空间问题排查

问题 #

出于教学目的，尝试通过ingress-nginx插件暴露kubernetes-dashboard。相关配置及排查信息如下：

kubernetes-dashboard-ingress.yaml配置 #

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dashboard-ingress
  namespace: kubernetes-dashboard
spec:
  rules:
    - host: dashboard.lc
      http:
        paths:
          - backend:
              service:
                name: kubernetes-dashboard
                port:
                  number: 80
            pathType: Prefix
            path: /

集群资源状态 #

• Pod状态（正常运行）：

$ kubectl get pods -n kubernetes-dashboard
NAME                                        READY   STATUS    RESTARTS   AGE
dashboard-metrics-scraper-5c6664855-fksqb   1/1     Running   0          16m
kubernetes-dashboard-55c4cbbc7c-6v6jt       1/1     Running   0          16m

• Service详情（服务存在于kubernetes-dashboard命名空间）：

$ kubectl get service kubernetes-dashboard -n kubernetes-dashboard -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","k8s-app":"kubernetes-dashboard","kubernetes.io/minikube-addons":"dashboard"},"name":"kubernetes-dashboard","namespace":"kubernetes-dashboard"},"spec":{"ports":[{"port":80,"targetPort":9090}],"selector":{"k8s-app":"kubernetes-dashboard"}}}
  creationTimestamp: "2023-03-08T08:50:51Z"
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: kubernetes-dashboard
    kubernetes.io/minikube-addons: dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
  resourceVersion: "30326"
  uid: a4a9d00e-88ba-4ad3-9027-42a34f7e9e68
spec:
  clusterIP: 10.101.22.189
  clusterIPs:
  - 10.101.22.189
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 80
    protocol: TCP
    targetPort: 9090
  selector:
    k8s-app: kubernetes-dashboard
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

$ kubectl get services -n kubernetes-dashboard
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
dashboard-metrics-scraper   ClusterIP   10.107.4.195    <none>        8000/TCP   14m
kubernetes-dashboard        ClusterIP   10.101.22.189   <none>        80/TCP     14m

• Ingress已成功应用：

$ kubectl get ingress -n kubernetes-dashboard
NAME                CLASS   HOSTS          ADDRESS        PORTS   AGE
dashboard-ingress   nginx   dashboard.lc   192.168.49.2   80      93m

异常现象 #

curl访问192.168.49.2返回503错误，查看ingress-nginx控制器日志（kubectl logs ingress-nginx-controller-77669ff58-fprp4 -n ingress-nginx）显示：

controller.go:948] Error creating upstream "default-kubernetes-dashboard-80": no object matching key "default/kubernetes-dashboard" in local store

controller.go:954] Error obtaining Service "default/kubernetes-dashboard": no object matching key "default/kubernetes-dashboard" in local store

Ingress控制器一直在default命名空间查找kubernetes-dashboard服务，但该服务实际位于kubernetes-dashboard命名空间，Ingress配置已指定命名空间，无法定位问题原因。

解决方案 #

1. 确认IngressClass关联有效性 #

先检查集群中是否存在nginx类型的IngressClass：

kubectl get ingressclass

如果没有，创建对应的IngressClass配置：

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx
spec:
  controller: k8s.io/ingress-nginx

执行命令应用：

kubectl apply -f ingressclass.yaml

2. 显式指定服务命名空间与IngressClass #

修改Ingress配置，显式指定服务所在命名空间及IngressClass（Kubernetes 1.22+支持跨命名空间服务引用）：

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dashboard-ingress
  namespace: kubernetes-dashboard
spec:
  ingressClassName: nginx
  rules:
    - host: dashboard.lc
      http:
        paths:
          - backend:
              service:
                name: kubernetes-dashboard
                namespace: kubernetes-dashboard
                port:
                  number: 80
            pathType: Prefix
            path: /

应用更新后的配置：

kubectl apply -f kubernetes-dashboard-ingress.yaml

3. 重启Ingress-Nginx控制器刷新缓存 #

强制重启控制器Pod，清除本地缓存的服务信息：

kubectl rollout restart deployment ingress-nginx-controller -n ingress-nginx

4. 验证控制器权限 #

检查Ingress-Nginx的ServiceAccount是否有权限访问kubernetes-dashboard命名空间的Service资源：

kubectl auth can-i get services -n kubernetes-dashboard --as=system:serviceaccount:ingress-nginx:ingress-nginx

如果返回no，添加对应的RBAC权限：

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ingress-nginx-cross-namespace
rules:
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ingress-nginx-cross-namespace
subjects:
- kind: ServiceAccount
  name: ingress-nginx
  namespace: ingress-nginx
roleRef:
  kind: ClusterRole
  name: ingress-nginx-cross-namespace
  apiGroup: rbac.authorization.k8s.io

应用权限配置：

kubectl apply -f ingress-nginx-rbac.yaml

5. 验证访问效果 #

修改本地hosts文件，添加192.168.49.2 dashboard.lc，然后访问http://dashboard.lc，或通过curl指定Host头测试：

curl -H "Host: dashboard.lc" 192.168.49.2

内容的提问来源于stack exchange，提问作者eksr