如何在Alertmanager中为HostOutOfMemory规则排除VM1作业

现有配置 #

Alertmanager告警规则 #

- alert: HostOutOfMemory
  expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10) * on(instance) group_left (nodename) node_uname_info{nodename=~".+"}
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: Host out of memory (instance {{ $labels.instance }})
    description: "Node memory is filling up (< 10% left)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

Prometheus采集配置 #

- job_name: "VM1"
  scrape_interval: 5s
  static_configs:
  - targets: ["192.168.0.24:9100"]
- job_name: "VM2"
  scrape_interval: 5s
  static_configs:
  - targets: ["192.168.0.25:9100"]

解决方案 #

方法1：修改告警规则表达式直接过滤 #

Prometheus采集的所有指标都会携带job标签，对应配置中的job_name。你可以在告警规则的表达式中添加job!="VM1"的过滤条件，直接排除VM1作业的实例：

修改后的expr字段如下：

expr: (node_memory_MemAvailable_bytes{job!="VM1"} / node_memory_MemTotal_bytes{job!="VM1"} * 100 < 10) * on(instance) group_left (nodename) node_uname_info{nodename=~".+", job!="VM1"}

或者更简洁的写法，利用指标关联后的标签继承，只在关联的up指标中过滤：

expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10) * on(instance) group_left (nodename) node_uname_info{nodename=~".+"} * on(instance) group_left(job) up{job!="VM1"}

方法2：在Alertmanager中配置抑制规则 #

如果不想改动Prometheus的告警规则，可以在Alertmanager配置里添加抑制规则，屏蔽来自VM1作业的HostOutOfMemory告警：

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'web.hook'
receivers:
- name: 'web.hook'
  webhook_configs:
  - url: 'http://example.com/webhook'
inhibit_rules:
- source_match:
    job: "VM1"
  target_match:
    alertname: "HostOutOfMemory"
  equal: ['instance']

这条规则会匹配所有来自VM1作业的告警，进而抑制同一实例上的HostOutOfMemory告警。

内容的提问来源于stack exchange，提问作者user13824551