Prometheus Adapter无法拉取自定义指标用于HPA配置问题

环境与采集配置 #

通过Prometheus社区Helm包安装Prometheus Operator与Prometheus Adapter，使用additionalscrapeconfig采集SpringBoot演示应用的指标，采集配置如下：

- job_name: "appops-springboot-hpa-demo-app"
  metrics_path: /actuator/prometheus
  static_configs:
    - targets: ["springboot-hpa-app.default.svc.cluster.local:8080"]

http_server_requests_seconds_count指标已在Prometheus中正常显示，但无法通过Adapter拉取。

尝试过的无效配置方案 #

方案1 #

- seriesQuery: 'http_server_requests_seconds_count{namespace !="",pod!="",uri="/requests"}'
      seriesFilters: []
      resources:
        overrides:
          namespace:
            resource: namespace
          pod:
            resource: pod
      name:
        matches: ""
        as: ""
      metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)'

方案2 #

custom:
    - seriesQuery: '{__name__=~"^http_server_requests_seconds_.*"}' 
      resources:
        overrides:
          kubernetes_namespace:
            resource: namespace
          kubernetes_pod_name:
            resource: pod
      name:
        matches: "^http_server_requests_seconds_count(.*)" 
        as: "http_server_requests_seconds_count_sum" 
      metricsQuery: sum ( <<.Series>> { <<.LabelMatchers>>,uri=~ "/requests.*" } ) by ( <<.GroupBy>> )

方案3 #

- seriesQuery: 'http_server_requests_seconds_count{job="appops-springboot-hpa-demo-app", method="GET", status="200", uri="/requests"}' 
      resources:
        overrides:
          kubernetes_namespace:
            resource: namespace
          kubernetes_pod_name:
            resource: pod
      name:
        matches: http_server_requests_seconds_count 
        as: "http_server_requests_seconds_count_sum" 
      metricsQuery: rate(http_server_requests_seconds_count{job="appops-springboot-hpa-demo-app", method="GET", status="200", uri="/requests",<<.LabelMatchers>>}[5m]) 

错误现象 #

调用自定义指标API：

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/*/http_server_requests_seconds_count | jq .

返回错误：

Error from server (NotFound): the server could not find the requested resource

Prometheus Adapter日志：

I0211 00:58:38.846373       1 httplog.go:104] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1/namespaces/ops-poc/pods/%2A/http_server_requests_seconds_count?labelSelector=app%3Dspringboot-custom-hpa" latency="15.099215ms" userAgent="kube-controller-manager/v1.22.13 (linux/amd64) kubernetes/a43c090/system:serviceaccount:kube-system:horizontal-pod-autoscaler" audit-ID="351e31aa-c8c9-4558-bb33-f265638ef678" srcIP="10.42.15.0:57042" resp=404

注：Adapter可正常拉取Helm默认指标，连通性无问题。

排查与修复建议 #

1. 注入K8s元标签到采集指标
   由于使用static_configs采集，Prometheus不会自动为指标添加namespace/pod这类K8s资源标签，Adapter无法关联到K8s资源。需修改采集配置，添加relabel_configs注入标签：

   - job_name: "appops-springboot-hpa-demo-app"
     metrics_path: /actuator/prometheus
     static_configs:
       - targets: ["springboot-hpa-app.default.svc.cluster.local:8080"]
     relabel_configs:
       # 从target地址提取namespace
       - source_labels: [__address__]
         regex: 'springboot-hpa-app\.([^.]+)\.svc\.cluster\.local:8080'
         target_label: namespace
       # 从target地址提取pod名称（如果应用pod名称固定为springboot-hpa-app）
       - target_label: pod
         replacement: "springboot-hpa-app"
       # 若应用自身暴露了kubernetes_pod_name标签，可映射为pod
       - action: labelmap
         regex: 'kubernetes_pod_name'
         replacement: 'pod'
   

2. 修正Adapter配置中的模板转义问题
   所有配置中的<<.Series>>是HTML转义字符，需改为原生模板语法<<.Series>>，否则Adapter无法解析变量。

3. 调整Adapter配置的匹配逻辑
   使用如下修正后的Adapter配置，确保标签匹配、资源映射和指标查询逻辑正确：

   - seriesQuery: 'http_server_requests_seconds_count{namespace!="", pod!="", uri="/requests"}'
     resources:
       overrides:
         namespace: {resource: "namespace"}
         pod: {resource: "pod"}
     name:
       matches: "^http_server_requests_seconds_count$"
       as: "http_server_requests_seconds_count_rate"
     metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)'
   

4. 验证配置生效与指标可用性

   • 修改Adapter ConfigMap后，重启Adapter pod：kubectl rollout restart deployment prometheus-adapter -n <your-namespace>
   • 查看所有可用自定义指标：kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq .
   • 调用正确的API路径（对应as字段的指标名）：

     kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_server_requests_seconds_count_rate | jq .
     

内容的提问来源于stack exchange，提问作者Sundar