Prometheus Alertmanager未触发Telegraf procstat的MySQL/Nginx PID告警

环境版本 #

Prometheus 2.39.0
Alertmanager 0.24.0
Telegraf 1.20.2

已配置内容 #

Prometheus采集任务 #

MySQL任务

- job_name: "gm_mysql_pid"
  scheme: "https"
  tls_config:
    insecure_skip_verify: true
  static_configs:
    - targets:
        [
          "host1.com:9273",
        ]

Nginx任务

- job_name: "gm_telegraf_exporter"
  scheme: "https"
  tls_config:
    insecure_skip_verify: true
  static_configs:
    - targets:
        [
          "host1.com:9273",
        ]

Alertmanager配置 #

route:
  group_by: ["alertname", "group", "instance"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 120h
  receiver: devops-team
  routes:
    - match:
        group: gm_telegraf_exporter
      continue: true
      receiver: prometheus-receiver
receivers:
 - name: "prometheus-receiver"
    slack_configs:
      - api_url: https://hooks.slack.com/xxx
        channel: "alerts-prometheus"
        send_resolved: true
        title: '{{ template "custom_title" . }}'
        text: '{{ template "custom_slack_message" . }}'

告警规则配置 #

# MySQL告警规则
- alert: gm_mysql_pid
  expr: procstat_lookup_pid_count{job="gm_mysql_pid",pid_finder="",pidfile="/var/run/mysqld/mysqld.pid"} <= 0
  labels:
    group: "gm_mysql_pid"
  annotations:
    identifier: "Host: {{$labels.host}}"
    description: "Trigger: MySQL service is down!"

# Nginx告警规则
- alert: gm_nginx_pid
  expr: procstat_lookup_pid_count{job="gm_telegraf_exporter",pid_finder="",pidfile="/var/run/nginx.pid"} <= 0
  labels:
    group: "gm_telegraf_exporter"
  annotations:
    identifier: "Host: {{$labels.host}}"
    description: "Trigger: Nginx service is down!"

正常状态指标 #

procstat_lookup_pid_count{host="host1.com",pid_finder="pgrep",pidfile="/var/run/mysqld/mysqld.pid",result="success"} 1
procstat_lookup_pid_count{host="host1.com",pid_finder="pgrep",pidfile="/var/run/nginx.pid",result="success"} 1

问题现象 #

停止MySQL和Nginx服务后，指标显示服务宕机，但Alertmanager未触发告警，此时指标为：

procstat_lookup_pid_count{host="host1.com",pid_finder="pgrep",pidfile="",result="lookup_error"} 0

问题原因及修复方案 #

1. 告警规则表达式不匹配 #

你的告警规则指定了pid_finder=""和具体pidfile，但服务宕机后生成的指标是pid_finder="pgrep"、pidfile=""，标签匹配条件不满足，导致规则无法触发。

修复后的告警规则

# MySQL告警规则
- alert: gm_mysql_pid
  expr: (procstat_lookup_pid_count{job="gm_mysql_pid",pidfile="/var/run/mysqld/mysqld.pid"} == 0) OR (procstat_lookup_pid_count{job="gm_mysql_pid",result="lookup_error"} == 0)
  labels:
    group: "gm_mysql_pid"
  annotations:
    identifier: "Host: {{$labels.host}}"
    description: "Trigger: MySQL service is down!"

# Nginx告警规则
- alert: gm_nginx_pid
  expr: (procstat_lookup_pid_count{job="gm_telegraf_exporter",pidfile="/var/run/nginx.pid"} == 0) OR (procstat_lookup_pid_count{job="gm_telegraf_exporter",result="lookup_error"} == 0)
  labels:
    group: "gm_telegraf_exporter"
  annotations:
    identifier: "Host: {{$labels.host}}"
    description: "Trigger: Nginx service is down!"

2. Alertmanager路由配置遗漏 #

当前路由仅匹配group: gm_telegraf_exporter的告警发送到Slack，MySQL告警的group: gm_mysql_pid只会发送到未配置接收方式的devops-team，即使触发也收不到通知。

修复后的Alertmanager路由（二选一即可）

方案一：修改默认接收者为已配置的prometheus-receiver

route:
  group_by: ["alertname", "group", "instance"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 120h
  receiver: prometheus-receiver
  routes:
    - match:
        group: gm_telegraf_exporter
      continue: true
      receiver: prometheus-receiver
receivers:
 - name: "prometheus-receiver"
    slack_configs:
      - api_url: https://hooks.slack.com/xxx
        channel: "alerts-prometheus"
        send_resolved: true
        title: '{{ template "custom_title" . }}'
        text: '{{ template "custom_slack_message" . }}'

方案二：新增MySQL告警的路由匹配规则

route:
  group_by: ["alertname", "group", "instance"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 120h
  receiver: devops-team
  routes:
    - match:
        group: gm_telegraf_exporter
      continue: true
      receiver: prometheus-receiver
    - match:
        group: gm_mysql_pid
      continue: true
      receiver: prometheus-receiver
receivers:
 - name: "prometheus-receiver"
    slack_configs:
      - api_url: https://hooks.slack.com/xxx
        channel: "alerts-prometheus"
        send_resolved: true
        title: '{{ template "custom_title" . }}'
        text: '{{ template "custom_slack_message" . }}'

3. 额外检查点 #

• 确认Prometheus已加载最新告警规则：访问http://<prometheus-ip>:9090/rules查看规则状态
• 确认Alertmanager配置生效：访问http://<alertmanager-ip>:9093/#/status查看路由配置
• 检查Prometheus采集状态：访问http://<prometheus-ip>:9090/targets确认gm_mysql_pid和gm_telegraf_exporter任务正常运行

内容的提问来源于stack exchange，提问作者Roberto Jobet