EKS集群中Prometheus Adapter Helm Chart未授权问题求助

我们在EKS集群中通过prometheus-community Helm Chart部署了Prometheus，为实现HPA基于CPU、内存及默认Metrics Server不支持的网络活动进行扩缩容，执行了以下部署prometheus-adapter的命令：

helm install custom-metrics prometheus-community/prometheus-adapter --namespace monitoring --set prometheus.url=http://prometheus-operated.monitoring.svc --set hostNetwork.enabled=true

部署成功后，kubectl get apiservices显示custom.metrics.k8s.io/v1beta1状态为True，但执行任意kubectl命令（如kubectl get pods -n monitoring）时，首行始终返回以下错误：

E0726 13:18:37.044670   28536 memcache.go:287] couldn't get resource list for custom.metrics.k8s.io/v1beta1: Unauthorized

后续命令结果可正常返回。执行kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 -n monitoring则提示：

Error from server (NotFound): the server could not find the requested resource

我们尝试过切换hostNetwork.enabled为true或默认false，均未解决问题。已知需为HPA创建自定义规则，但需先解决适配器运行问题，求配置修改方案。

当前SVC查询结果：

NAME                                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
alertmanager-operated                            ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   21d
custom-metrics-prometheus-adapter                ClusterIP   172.20.67.162    <none>        443/TCP                      82m
kube-prometheus-stack-alertmanager               ClusterIP   172.20.196.135   <none>        9093/TCP                     21d
kube-prometheus-stack-grafana                    ClusterIP   172.20.175.12    <none>        80/TCP                       21d
kube-prometheus-stack-kube-state-metrics         ClusterIP   172.20.20.63     <none>        8080/TCP                     21d
kube-prometheus-stack-operator                   ClusterIP   172.20.145.105   <none>        443/TCP                      21d
kube-prometheus-stack-prometheus                 ClusterIP   172.20.178.62    <none>        9090/TCP                     21d
kube-prometheus-stack-prometheus-node-exporter   ClusterIP   172.20.217.223   <none>        9100/TCP                     21d
prometheus-operated                              ClusterIP   None             <none>        9090/TCP                     21d

针对EKS环境下prometheus-adapter的权限和资源发现问题，按以下步骤排查修复：

1. 修复RBAC权限配置 #

Unauthorized错误根源是kube-apiserver访问prometheus-adapter时缺少正确的RBAC权限，按以下步骤调整：

• 查看当前prometheus-adapter的ServiceAccount：

  kubectl get sa custom-metrics-prometheus-adapter -n monitoring
  

• 检查对应的ClusterRole和ClusterRoleBinding是否存在：

  kubectl get clusterrole | grep prometheus-adapter
  kubectl get clusterrolebinding | grep prometheus-adapter
  

• 如果权限缺失，手动创建包含必要权限的ClusterRole：

  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    name: custom-metrics-prometheus-adapter
  rules:
  - apiGroups: ["custom.metrics.k8s.io"]
    resources: ["*"]
    verbs: ["*"]
  - apiGroups: ["metrics.k8s.io"]
    resources: ["*"]
    verbs: ["*"]
  - apiGroups: [""]
    resources: ["pods", "nodes", "namespaces"]
    verbs: ["get", "list", "watch"]
  

  绑定到目标ServiceAccount：

  kubectl create clusterrolebinding custom-metrics-prometheus-adapter --clusterrole=custom-metrics-prometheus-adapter --serviceaccount=monitoring:custom-metrics-prometheus-adapter
  

2. 验证Prometheus访问连通性 #

当前配置的prometheus.url=http://prometheus-operated.monitoring.svc可能存在网络策略或Prometheus自身权限限制：

• 进入prometheus-adapter Pod验证访问：

  kubectl exec -n monitoring <prometheus-adapter-pod-name> -- curl http://prometheus-operated.monitoring.svc:9090/api/v1/query?query=up
  

• 如果访问失败，添加允许适配器访问Prometheus的网络策略：

  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    name: allow-prometheus-adapter-access
    namespace: monitoring
  spec:
    podSelector:
      matchLabels:
        app.kubernetes.io/name: prometheus
    ingress:
    - from:
      - podSelector:
          matchLabels:
            app.kubernetes.io/name: prometheus-adapter
      ports:
      - protocol: TCP
        port: 9090
  

3. 配置自定义Metrics暴露规则 #

创建values.yaml文件覆盖Helm配置，定义网络相关Metrics的暴露规则：

prometheus:
  url: http://prometheus-operated.monitoring.svc
  port: 9090
rules:
  default: false
  custom:
  - seriesQuery: 'container_network_receive_bytes_total{namespace!="",pod!=""}'
    resources:
      overrides:
        namespace: {resource: "namespace"}
        pod: {resource: "pod"}
    name:
      matches: "^(container_network_receive_bytes_total)$"
      as: "pod_network_receive_bytes_per_second"
    metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)'
  - seriesQuery: 'container_network_transmit_bytes_total{namespace!="",pod!=""}'
    resources:
      overrides:
        namespace: {resource: "namespace"}
        pod: {resource: "pod"}
    name:
      matches: "^(container_network_transmit_bytes_total)$"
      as: "pod_network_transmit_bytes_per_second"
    metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)'

升级Helm部署：

helm upgrade custom-metrics prometheus-community/prometheus-adapter --namespace monitoring -f values.yaml

4. 验证修复效果 #

• 重启prometheus-adapter Pod：

  kubectl rollout restart deployment custom-metrics-prometheus-adapter -n monitoring
  

• 查询自定义Metrics状态：

  kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq
  

  若能返回Metrics列表，说明配置生效，此时执行kubectl命令的Unauthorized错误应消失。

内容的提问来源于stack exchange，提问作者Boonya