如何配置Prometheus Alertmanager减少高CPU负载重复告警？

我配置了Prometheus的HostHighCpuLoad95告警规则，当前会收到多条重复的告警通知，示例如下：

• CPU load is > 95% VALUE = 97.23
• CPU load is > 95% VALUE = 96.23
• CPU load is > 95% VALUE = 98.23
• CPU load is > 95% VALUE = 94.23

尽管已将repeat_interval配置为24小时，但每条告警仍会每24小时重复发送，且期间只要CPU负载数值变化就会收到新的告警。我的需求是：在CPU负载降至94%以下前，仅需每24小时接收一次该类告警，停止接收额外的重复通知。

现有配置 #

Alertmanager配置 (alertmanager.conf) #

global:
  resolve_timeout: 1440m
  telegram_api_url: "https://api.telegram.org"
  http_config:
    follow_redirects: true

templates:
  - '/opt/prometheus/etc/rules/alertmanager_notifications_telegram.tmpl'

route:
  group_by: ['instance']
  group_wait: 60s
  group_interval: 5s
  repeat_interval: 719h
  receiver: 'emaildevelopers'

  routes:
  - receiver: 'telegram'
    group_by: [instance, alertname, service]
    group_wait: 10s
    repeat_interval: 24h
    continue: true
    matchers:
    - severity="critical"
  
  - receiver: 'emailteamworker'
    group_by: [instance]
    group_wait: 10s
    repeat_interval: 24h
    continue: true
    matchers:
    - receiver="email"
  
inhibit_rules:
- equal: ['instance']
  source_matchers:
    - severity = critical
  target_matchers:
    - "severity =~ warning|info"

receivers:
  - name: 'telegram'
    telegram_configs:
    - api_url: 'https://api.telegram.org'
    ...
  
  - name: 'emailteamworker'
    email_configs:
    - to: 'xxx@tasks.teamwork.com'
      from: 'XX YY <devops@domain.com>'
      smarthost: 'smtp.sendgrid.net:587'
      hello: 'domain.com'
      auth_username: 'apikey'
      auth_password: 'SG.cvfg'
      send_resolved: false
      headers:
    ...

Prometheus告警规则配置 #

- name: Host and hardware node-exporter
  ...
    - alert: HostHighCpuLoad95
      expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[10m])) * 100) > 95
      for: 1m
      labels:
        severity: critical
        receiver: email
      annotations:
        summary: "Host high CPU load > 95% instance {{ $labels.instance }} for last 60 minutes"
        description: "CPU load is > 95%  VALUE = {{ $value | humanize }}  LABELS = {{ $labels }}"
  ...

问题出在group_interval的配置上：当前父路由的group_interval设为5秒，且子路由未覆盖该值。group_interval的作用是当告警组内已有告警发送后，若组内告警状态更新（比如CPU负载数值变化），需等待多久再次发送该组的告警通知。5秒的间隔会导致只要负载数值变动，就会频繁触发新的通知，完全忽略了repeat_interval的限制。

修改步骤如下：

1. 在对应的子路由中添加group_interval配置，将其设置为与repeat_interval相同的24小时，确保在一个周期内仅发送一次告警通知，即使负载数值有变化。

修改后的Alertmanager子路由配置：

routes:
- receiver: 'telegram'
  group_by: [instance, alertname, service]
  group_wait: 10s
  group_interval: 24h  # 添加这一行
  repeat_interval: 24h
  continue: true
  matchers:
  - severity="critical"

- receiver: 'emailteamworker'
  group_by: [instance]
  group_wait: 10s
  group_interval: 24h  # 添加这一行
  repeat_interval: 24h
  continue: true
  matchers:
  - receiver="email"

2. 验证告警分组逻辑：你的告警规则已经通过avg by(instance)按实例聚合，每个实例只会生成一条HostHighCpuLoad95告警，配合上述group_interval的修改，同一个实例的告警会被归为一组，仅每24小时发送一次通知，直到CPU负载降至95%以下（告警恢复）。

3. 可选：若希望告警恢复时不发送通知，当前配置的send_resolved: false已经满足需求，无需调整。

内容的提问来源于stack exchange，提问作者Rostyslav Malenko