私有GKE集群gke-metrics-agent向UAS导出指标权限错误求助

问题现象 #

在私有GKE集群（版本1.23.14-gke.1800）的kube-system命名空间中，gke-metrics-agent Pod日志出现如下权限错误：

error uasexporter/exporter.go:190 Error exporting metrics to UAS  {"kind": "exporter", "name": "uas", "error": "reading from stream failed: rpc error: code = PermissionDenied desc = The caller does not have permission"}

error   uasexporter/exporter.go:226 failed to get response from UAS {"kind": "exporter", "name": "uas", "error": "rpc error: code = PermissionDenied desc = The caller does not have permission"}

日志元信息：

• app: gke-metrics-agent
• component: gke-metrics-agent
• container: gke-metrics-agent
• filename: /var/log/pods/kube-system_gke-metrics-agent-9rbfv_6896b214-31d2-43bb-b15d-a8e1b122d41d/gke-metrics-agent/0.log
• job: kube-system/gke-metrics-agent
• namespace: kube-system
• node_name: gke-gke-production-production-88f13984-h83x
• pod: gke-metrics-agent-9rbfv
• stream: stderr

当前RBAC配置 #

gke-metrics-agent对应的ServiceAccount、ClusterRoleBinding及ClusterRole配置如下：

apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2022-12-07T10:20:55Z"
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
  name: gke-metrics-agent
  namespace: kube-system
  resourceVersion: "444"
  uid: ...
secrets: ..
- name: gke-metrics-agent-token-6zhvq

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2022-12-07T10:20:56Z"
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
  name: gke-metrics-agent
  resourceVersion: "452"
  uid: ...
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: gke-metrics-agent
subjects:
- kind: ServiceAccount
  name: gke-metrics-agent
  namespace: kube-system

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: "2022-12-07T10:20:56Z"
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
  name: gke-metrics-agent
  resourceVersion: "67979037"
  uid: ...
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - list
  - watch
- apiGroups:
  - policy
  resourceNames:
  - gce.gke-metrics-agent
  resources:
  - podsecuritypolicies
  verbs:
  - use

已尝试操作及关联问题 #

• gke-metrics-agent是GKE自动部署的官方DaemonSet，已确认是权限问题，但不清楚UAS的具体含义，在GCP文档及网络中未找到有效信息
• 尝试为该ServiceAccount授予system:gke-uas-metrics-reader、external-metrics-reader等额外ClusterRole，问题仍未解决
• 集群偶尔出现Kubernetes聚合API v1beta1.metrics.k8s.io/default报错，近10分钟仅75%可用，推测与上述权限错误相关

解决方向 #

1. 明确UAS的含义 #

UAS是GKE内部的Unified Agent Service，属于管控平面组件，负责接收gke-metrics-agent上报的节点、Pod监控指标，是GKE监控链路的核心环节之一。

2. 修复RBAC权限完整性 #

官方gke-metrics-agent的ClusterRole默认包含与UAS通信的权限，若当前配置缺失，可能是集群升级、自定义配置覆盖导致：

• 检查集群中是否存在system:gke-uas相关的ClusterRole或Role，确认gke-metrics-agent ServiceAccount是否绑定正确
• 使用gcloud container clusters update [CLUSTER_NAME] --update-addons=GkeMetricsAgent=ENABLED触发GKE重新同步组件的默认RBAC配置

3. 排查聚合API不可用问题 #

聚合API v1beta1.metrics.k8s.io依赖metrics-server或gke-metrics-agent提供指标数据：

• 检查metrics-server Pod状态（若集群启用），确认其运行正常且无权限错误
• 验证聚合层配置：检查kube-apiserver的--enable-aggregator-routing参数是否开启，确认聚合API的端点状态
• 查看kube-system命名空间中与聚合API相关的Pod（如metrics-server、gke-metrics-agent）的日志，定位具体报错

4. 重置gke-metrics-agent组件 #

若手动调整权限无效，可让GKE重新部署组件：

• 删除gke-metrics-agent DaemonSet：kubectl delete daemonset gke-metrics-agent -n kube-system，GKE的addon-manager会自动重建该组件
• 重建后检查新的RBAC配置是否包含UAS相关权限规则

5. 检查私有集群网络连通性 #

私有集群中节点与管控平面的网络连通性是UAS通信的前提：

• 确认集群的防火墙规则允许节点池的IP范围访问管控平面的443端口及UAS相关端口
• 检查节点是否能正常解析GKE管控平面的域名，确保DNS配置无误

内容的提问来源于stack exchange，提问作者Vesselin Yanev