Kubernetes/Helm Ingress部署失败，疑似权限问题求助

问题背景 #

所有Deployment及Service运行正常，但通过NGINX Ingress暴露服务时失败，控制器Pod启动后退出，日志显示权限及API兼容性问题。

我的Ingress Helm Chart模板 #

{{- if or (eq .Values.environment "staging") (eq .Values.environment "production") -}}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sparrow-{{ .Values.environment }}-ingress
  namespace: sparrow-{{ .Values.environment }}
  annotations:
{{/*    kubernetes.io/ingress.class: nginx*/}}
spec:
  ingressClassName: nginx
  rules:
{{- if eq .Values.environment "staging" }}
    - host: staging.sparrow.express
{{- end -}}
{{- if eq .Values.environment "production" }}
    - host: sparrow.express
{{- end }}
      http:
        paths:
          - path: /(.*)
            pathType: Prefix
            backend:
              service:
                name: {{ .Values.applicationName }}-ui-service
                port:
                  number: 80
          - path: /api/(.*)
            pathType: Prefix
            backend:
              service:
                name: {{ .Values.applicationName }}-api-service
                port:
                  number: 3000
{{- end -}}

控制器Pod启动日志 #

I0626 00:58:55.103267       7 flags.go:205] Watching for Ingress class: nginx
W0626 00:58:55.103505       7 flags.go:250] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W0626 00:58:55.103555       7 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0626 00:58:55.103848       7 main.go:231] Creating API client for https://10.96.0.1:443
I0626 00:58:55.107562       7 main.go:275] Running in Kubernetes cluster version v1.26 (v1.26.3) - git (clean) commit 9e644106593f3f4aa98f8a84b23db5fa378900bd - platform linux/arm64
I0626 00:58:55.109148       7 main.go:87] Validated default/sparrow-staging-nginx-ingress-default-backend as the default backend.
I0626 00:58:55.202268       7 main.go:105] SSL fake certificate created /etc/ingress-controller/ssl/default-fake-certificate.pem
I0626 00:58:55.202908       7 main.go:113] Enabling new Ingress features available since Kubernetes v1.18
E0626 00:58:55.203546       7 main.go:122] Unexpected error searching IngressClass: ingressclasses.networking.k8s.io "nginx" is forbidden: User "system:serviceaccount:default:sparrow-staging-nginx-ingress" cannot get resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
W0626 00:58:55.203558       7 main.go:125] No IngressClass resource with name nginx found. Only annotation will be used.
W0626 00:58:55.211518       7 store.go:659] Unexpected error reading configuration configmap: configmaps "sparrow-staging-nginx-ingress-controller" not found
I0626 00:58:55.215679       7 nginx.go:263] Starting NGINX Ingress controller
E0626 00:58:56.320559       7 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource
E0626 00:58:57.775395       7 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource
E0626 00:58:59.873440       7 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource
E0626 00:59:05.758255       7 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource
E0626 00:59:17.059505       7 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource
I0626 00:59:30.946386       7 main.go:179] Received SIGTERM, shutting down
I0626 00:59:30.946406       7 nginx.go:380] Shutting down controller queues
I0626 00:59:30.946421       7 status.go:118] updating status of Ingress rules (remove)
E0626 00:59:30.946546       7 store.go:186] timed out waiting for caches to sync
I0626 00:59:30.946573       7 nginx.go:307] Starting NGINX process
I0626 00:59:30.946761       7 leaderelection.go:242] attempting to acquire leader lease  default/ingress-controller-leader-nginx...
E0626 00:59:30.946908       7 queue.go:78] queue has been shutdown, failed to enqueue: &ObjectMeta{Name:initial-sync,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[]OwnerReference{},Finalizers:[],ClusterName:,ManagedFields:[]ManagedFieldsEntry{},}
I0626 00:59:30.953230       7 leaderelection.go:252] successfully acquired lease default/ingress-controller-leader-nginx
E0626 00:59:30.953382       7 queue.go:78] queue has been shutdown, failed to enqueue: &ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[]OwnerReference{},Finalizers:[],ClusterName:,ManagedFields:[]ManagedFieldsEntry{},}
I0626 00:59:30.953435       7 status.go:86] new leader elected: sparrow-staging-nginx-ingress-controller-74555f4f44-d77f7
I0626 00:59:30.957187       7 status.go:137] removing address from ingress status ([192.168.49.2])
I0626 00:59:30.957328       7 nginx.go:396] Stopping NGINX process
2023/06/26 00:59:30 [notice] 26#26: signal process started
I0626 00:59:33.968556       7 nginx.go:409] NGINX process has stopped
I0626 00:59:33.968592       7 main.go:187] Handled quit, awaiting Pod deletion
I0626 00:59:43.969787       7 main.go:190] Exiting with 0

关键问题分析 #

从日志中提取三个核心故障点：

1. 权限不足：服务账号system:serviceaccount:default:sparrow-staging-nginx-ingress无集群级ingressclasses资源的访问权限
2. API版本不兼容：控制器尝试调用已废弃的networking.k8s.io/v1beta1/Ingress API，而Kubernetes 1.26已移除该版本支持
3. 必要资源缺失：控制器依赖的ConfigMapsparrow-staging-nginx-ingress-controller不存在

修复步骤 #

1. 升级NGINX Ingress Controller版本 #

Kubernetes 1.22+已全面使用networking.k8s.io/v1作为Ingress的稳定API版本，废弃了v1beta1。需将控制器升级到v1.0.0及以上版本（对应Helm Chart版本v4.0.0+），新版本完全适配Kubernetes 1.26的API规范。

2. 补全服务账号集群权限 #

创建ClusterRole并绑定到控制器的服务账号，赋予其访问IngressClass等集群资源的权限：

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
- apiGroups: ["networking.k8s.io"]
  resources: ["ingressclasses", "ingresses"]
  verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: [""]
  resources: ["services", "endpoints", "pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrolebinding
subjects:
- kind: ServiceAccount
  name: sparrow-staging-nginx-ingress
  namespace: default
roleRef:
  kind: ClusterRole
  name: nginx-ingress-clusterrole
  apiGroup: rbac.authorization.k8s.io

若使用Helm安装控制器，确保values.yaml中rbac.create=true，默认Chart会自动生成所需的RBAC资源。

3. 创建缺失的IngressClass资源 #

检查集群中是否存在名为nginx的IngressClass：

kubectl get ingressclasses

若不存在，创建该资源：

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx
spec:
  controller: k8s.io/ingress-nginx

4. 创建控制器所需的ConfigMap #

控制器依赖指定名称的ConfigMap加载配置，创建示例ConfigMap：

apiVersion: v1
kind: ConfigMap
metadata:
  name: sparrow-staging-nginx-ingress-controller
  namespace: default
data:
  ssl-protocols: "TLSv1.2 TLSv1.3"
  use-forwarded-headers: "true"

若使用Helm，确保controller.config.enabled=true，Chart会自动生成该ConfigMap。

5. 验证Ingress资源部署 #

部署后检查Ingress资源状态：

kubectl get ingress -n sparrow-staging
kubectl describe ingress sparrow-staging-ingress -n sparrow-staging

总结 #

核心故障是控制器版本与Kubernetes版本不匹配，叠加权限配置缺失和必要资源未创建。按上述步骤修复后，控制器可正常启动并处理Ingress规则。

内容的提问来源于stack exchange，提问作者jonathan-lee-devel