如何在Alertmanager中配置基于时间的磁盘使用率告警升级策略？

已配置磁盘使用率分层告警：

• 使用率达85%时触发告警通知admin
• 90%时通知super-admin
• 95%时通知ultra-admin

当前遇到问题：当85%告警触发后，磁盘使用率维持在88%且超过40分钟未被admin处理，需在Alertmanager中配置基于时间的升级策略：

• 告警30分钟未解决 → 升级通知super-admin
• 告警45分钟未解决 → 升级通知ultra-admin

原85%告警规则（90%、95%为同类配置）：

- alert: DiskUsage85
    expr: '(100 - (sum(node_filesystem_avail_bytes{instance="abc.xyz:9100"}  / node_filesystem_size_bytes{instance="abc.xyz:9100"}) BY (device,instance,mountpoint,stack) * 100)) > 85'
    for: 5m
    labels:
      maildest: admin
      severity: critical
    annotations:
      description: '{{reReplaceAll "(.*):(.*)" "${1}" $labels.instance}} - Disk {{$labels.device}} ({{$labels.mountpoint}}) has {{$value | printf "%.2f"}} (>80%) space usage'
      summary: 'WARN: {{ reReplaceAll "(.*):(.*)" "${1}" $labels.instance }} - Disk {{$labels.device }} ({{$labels.mountpoint}}) is {{$value | printf "%.2f"}}% full'

原Alertmanager基础配置：

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 24h
  receiver: team-X-mails
  
  routes:
    - receiver: admin
      match:
        maildest: admin

    - receiver: super-admin
      match:
        maildest: super-admin

    - receiver: ultra-admin
      match:
        maildest: ultra-admin
        
receivers:
- name: 'team-X-mails'
  email_configs:
    - to: 'abc@gmail.com'
      send_resolved: true
      headers:
        subject: '[{{ .Status | toUpper }}{{if eq .Status "firing"}}:{{.Alerts.Firing|len}}{{end}}] {{ .CommonAnnotations.summary }}'

- name: 'admin'
  email_configs:
    - to: 'admin@gmail.com'
      send_resolved: true
      headers:
        subject: '[{{ .Status | toUpper }}{{if eq .Status "firing"}}:{{.Alerts.Firing|len}}{{end}}] {{ .CommonAnnotations.summary }}'

核心利用Alertmanager路由的wait_for参数实现延迟升级通知，通过continue: true保留告警的递进匹配逻辑，同时兼容原有分层告警规则。

1. 补充缺失的Receiver配置 #

先在receivers节点中添加super-admin和ultra-admin的邮件配置：

- name: 'super-admin'
  email_configs:
    - to: 'super-admin@gmail.com'
      send_resolved: true
      headers:
        subject: '[{{ .Status | toUpper }}{{if eq .Status "firing"}}:{{.Alerts.Firing|len}}{{end}}] {{ .CommonAnnotations.summary }}'

- name: 'ultra-admin'
  email_configs:
    - to: 'ultra-admin@gmail.com'
      send_resolved: true
      headers:
        subject: '[{{ .Status | toUpper }}{{if eq .Status "firing"}}:{{.Alerts.Firing|len}}{{end}}] {{ .CommonAnnotations.summary }}'

2. 调整Alertmanager路由规则 #

修改route.routes节点，优先处理DiskUsage85的升级逻辑，再保留原有按maildest匹配的规则：

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 24h
  receiver: team-X-mails
  
  routes:
    # DiskUsage85告警升级策略
    - match:
        alertname: DiskUsage85
      receiver: admin
      continue: true  # 允许告警继续匹配后续路由
    
    - match:
        alertname: DiskUsage85
      receiver: super-admin
      wait_for: 30m  # 告警触发30分钟未解决，发送给super-admin
      continue: true
    
    - match:
        alertname: DiskUsage85
      receiver: ultra-admin
      wait_for: 45m  # 告警触发45分钟未解决，发送给ultra-admin
    
    # 原有分层告警匹配规则（对应90%、95%告警）
    - receiver: admin
      match:
        maildest: admin

    - receiver: super-admin
      match:
        maildest: super-admin

    - receiver: ultra-admin
      match:
        maildest: ultra-admin

配置说明 #

• wait_for：指定告警处于firing状态满设定时长后，才会发送给对应接收人，实现延迟升级
• continue: true：允许同一个告警继续匹配后续路由，保证升级逻辑的递进性
• 原有90%、95%的告警会通过maildest标签直接匹配对应接收人，不受升级策略影响
• send_resolved: true：在告警恢复时发送通知，便于管理员确认问题已解决

内容的提问来源于stack exchange，提问作者zimba