Prometheus采集Kubernetes Pod指标仅显示CPU内存两项问题排查

配置存在的核心问题 #

• 容器名过滤规则过严：配置中的source_labels: [__meta_kubernetes_pod_container_name] action: keep regex: prometheus.*规则会直接丢弃所有容器名不以prometheus开头的Pod采集任务，仅保留Prometheus相关组件的Pod数据。你当前能查到的Kube_pod_CPU、Kube_pod_memory属于kube-state-metrics暴露的集群元数据指标，并非本Job采集的业务Pod指标。
• 配置存在语法错误：/prometheus-kubernetes.yml属于冗余内容，错误插入到地址重写规则块中，会导致整个relabel配置解析失败，规则无法正常生效。
• 抓取注解匹配规则不规范：匹配prometheus.io/scrape注解的规则填写了regex: true，Prometheus正则匹配接收字符串类型值，需修改为regex: 'true'避免匹配异常。
• 地址重写正则兼容性不足：当前启用的(.+):(?:\d+);(\d+)正则在Pod地址未携带默认端口的场景下会匹配失败，建议启用注释的([^:]+)(?::\d+)?;(\d+)通用适配规则。

修复后配置参考 #

- job_name: 'kubernetes-pods'
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
   - role: pod
     namespaces:
       names: ["namespace-name"]
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    separator: ":"
    replacement: $1
    regex: 'true'
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    separator: ":"
    replacement: $1
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    separator: ":"
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: ${1}:${2}
    target_label: __address__
  # 如无需过滤容器名可注释或删除下方规则
  # - source_labels: [__meta_kubernetes_pod_container_name]
  #   action: keep
  #   regex: prometheus.*
  - action: labelmap
    separator: ":"
    replacement: $1
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    separator: ":"
    replacement: $1
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ":"
    replacement: $1
    action: replace
    target_label: kubernetes_pod_name

内容的提问来源于stack exchange，提问作者learning fun