Prometheus Adapter作为HPA自定义指标服务器时指标获取失败问题

环境信息 #

• monitoring命名空间：运行2个Prometheus Pod、1个Prometheus Adapter
• demo-config-app命名空间：部署待扩缩容的Spring Boot应用
• dynamic-secrets-ns命名空间：部署HPA配置

问题现象 #

HPA无法从Prometheus Adapter获取自定义指标，日志报错：

E0318 06:31:39.833494       1 provider.go:186] None of the results returned by when fetching metric services/http_server_requests_seconds_count(namespaced) for "dynamic-secrets-ns/demo-config-watcher-svc-internal" matched the resource name

相关配置 #

1. Prometheus Adapter ConfigMap自定义指标规则 #

rules:
    - seriesQuery: 'http_server_requests_seconds_count{namespace!="", service != "", uri = "/"}'
      resources: 
        overrides:
          namespace: {resource: "namespace"}
          service: {resource: "service"}
      name:
        matches: "http_server_requests_seconds_count"
        as: "http_server_requests_seconds_count"
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,uri!~"/actuator/.*"}[15m]))

2. HPA YAML清单 #

kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2
metadata:
  name: demo-config-app
  namespace: dynamic-secrets-ns
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: demo-config-watcher
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Object
    object:
      metric:
        name: http_server_requests_seconds_count
      describedObject:
        apiVersion: v1
        kind: Service
        name: demo-config-watcher-svc-internal
      target:
        type: AverageValue
        averageValue: 10

已完成的测试验证 #

• 执行kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta2" | jq，确认自定义指标已注册
• 在Prometheus控制台执行查询sum(rate(http_server_requests_seconds_count{namespace="dynamic-secrets-ns",service="demo-config-watcher-svc-internal",uri!~"/actuator/.*"}[15m]))，返回有效聚合值3.1471300541724765
• 同ServiceAccount的Nginx Pod可正常获取指标序列
• 已为HPA的ServiceAccount配置ClusterRole与ClusterRoleBinding，允许访问custom.metrics.k8s.io所有资源

问题根源与解决方案 #

根源1：Prometheus查询结果丢失关联标签 #

当前metricsQuery使用sum()聚合时未保留service和namespace标签，导致Prometheus Adapter无法将指标结果关联到对应的Kubernetes Service资源。

修复方案：修改Prometheus Adapter的metricsQuery，通过sum by (service, namespace)保留关联标签：

metricsQuery: sum by (service, namespace) (rate(<<.Series>>{<<.LabelMatchers>>,uri!~"/actuator/.*"}[15m]))

根源2：HPA未指定Service所在命名空间 #

HPA部署在dynamic-secrets-ns，但目标Service位于demo-config-app命名空间，未指定namespace时HPA会默认查询当前命名空间的Service，导致匹配失败。

修复方案：在HPA的describedObject中添加Service的命名空间：

describedObject:
  apiVersion: v1
  kind: Service
  name: demo-config-watcher-svc-internal
  namespace: demo-config-app

验证步骤 #

1. 更新Prometheus Adapter的ConfigMap配置，重启Adapter Pod使配置生效
2. 更新HPA配置清单
3. 执行kubectl describe hpa demo-config-app -n dynamic-secrets-ns查看HPA状态与事件，确认指标已正常获取

内容的提问来源于stack exchange，提问作者Mandar K