Prometheus remote_write过滤指标失效问题及反向过滤配置咨询

我正在监控kube-state-metrics的Pod指标，因为指标数量过多，想过滤部分指标不上传到Grafana。
我的remote_write配置如下：

remote_write:
- url: https://grafana-host:12345/api/prom/push
  basic_auth:
    username: myself
    password: somePassword
  metric_relabel_configs:
  - source_labels: [__name__,reason]
    regex: "kube_pod_container_status_last_terminated_reason;OOMKilled"
    action: keep

执行curl localhost:8080/metrics | grep kube_pod_container_status_last_terminated_reason查询Pod的/metrics端点，能看到该指标包含多种终止原因：

# HELP kube_pod_container_status_last_terminated_reason Describes the last reason the container was in terminated state.
# TYPE kube_pod_container_status_last_terminated_reason gauge
kube_pod_container_status_last_terminated_reason{...some other labels...,reason="OOMKilled"} 1
kube_pod_container_status_last_terminated_reason{...some other labels...,reason="Completed"} 1

但Grafana里还是能看到两类指标都被上报，不符合预期。另外想咨询如何配置反向过滤：保留所有reason标签不为"OOMKilled"的指标？

1. 原过滤配置失效的修复 #

你的metric_relabel_configs配置存在格式问题：YAML中无需转义双引号，同时要确保正则匹配的拼接逻辑正确（多标签默认用分号拼接值）。修复后的正确配置如下：

remote_write:
- url: https://grafana-host:12345/api/prom/push
  basic_auth:
    username: myself
    password: somePassword
  metric_relabel_configs:
  - source_labels: [__name__, reason]
    regex: 'kube_pod_container_status_last_terminated_reason;OOMKilled'
    action: keep

配置修改后，记得重新加载Prometheus配置：

curl -X POST http://prometheus-host:9090/-/reload

同时注意reason标签的大小写——Prometheus正则匹配是大小写敏感的，必须和指标中的标签值完全一致。

2. 反向过滤：保留非OOMKilled的指标 #

有两种实现方式：

方式一：丢弃OOMKilled的指标 #

直接匹配reason=OOMKilled的指标并丢弃，其余指标自动保留：

remote_write:
- url: https://grafana-host:12345/api/prom/push
  basic_auth:
    username: myself
    password: somePassword
  metric_relabel_configs:
  - source_labels: [__name__, reason]
    regex: 'kube_pod_container_status_last_terminated_reason;OOMKilled'
    action: drop

方式二：保留非OOMKilled的指标 #

使用负向前瞻正则，匹配所有reason不是OOMKilled的指标并保留：

remote_write:
- url: https://grafana-host:12345/api/prom/push
  basic_auth:
    username: myself
    password: somePassword
  metric_relabel_configs:
  - source_labels: [__name__, reason]
    regex: 'kube_pod_container_status_last_terminated_reason;(?!OOMKilled).+'
    action: keep

内容的提问来源于stack exchange，提问作者Avishay Cohen