Prometheus自定义告警FIRING时Alertmanager未推送邮件问题

问题背景 #

基于Prometheus技术栈实现自定义告警推送，通过prometheus-community helm chart部署Prometheus，同时部署Mailhog用于捕获Alertmanager发送的告警邮件。

• 部署后Prometheus自带的默认告警可正常推送至Mailhog
• 自定义PrometheusRule规则已在Prometheus Web UI中显示为FIRING触发状态，但Mailhog始终无法收到该自定义告警的邮件
• 已按指引为自定义PrometheusRule添加release: prometheus标签

已部署配置 #

AlertmanagerConfig资源 #

kind: AlertmanagerConfig
metadata:
  name: mailhog
  namespace: prometheus
spec:
  receivers:
  - name: mailhog
    emailConfigs:
    - to: "test@fake.com"
      from: "cluster@fake.com"
      smarthost: "mailhog.mailhog.svc.cluster.local:1025"
      requireTLS: false
  route:
    groupBy:
    - severity
    groupInterval: 30s
    groupWait: 30s
    receiver: mailhog
    repeatInterval: 1m

自定义PrometheusRule资源 #

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: prom-rule-test
  namespace: prometheus
  labels:
    release: prometheus
spec:
  groups:
    - name: ruleTest
      rules:
        - alert: test
          expr: vector(1)
          labels:
            severity: warning

Prometheus自定义资源 #

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  annotations:
    meta.helm.sh/release-name: prometheus
    meta.helm.sh/release-namespace: prometheus
    project.cattle.io/namespaces: '["aws-ecr-helper","aws-efs","cattle-system","cert-manager","fluentd","ingress-nginx","kube-node-lease","kube-public","kube-system","prometheus"]'
  creationTimestamp: "2022-07-07T08:57:27Z"
  generation: 5
  labels:
    app: kube-prometheus-stack-prometheus
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kube-prometheus-stack
    app.kubernetes.io/version: 36.2.1
    chart: kube-prometheus-stack-36.2.1
    heritage: Helm
    release: prometheus
  name: prometheus-kube-prometheus-prometheus
  namespace: prometheus
  resourceVersion: "225742469"
  uid: 68ce943f-95e4-4052-86f8-d095c4e0692b
spec:
  alerting:
    alertmanagers:
    - name: prometheus-kube-prometheus-alertmanager
      namespace: prometheus
      pathPrefix: /
      port: http-web
  evaluationInterval: 30s
  externalUrl: http://myurl.com/
  image: quay.io/prometheus/prometheus:v2.36.1
  logFormat: logfmt
  logLevel: info
  podMonitorNamespaceSelector: {}
  podMonitorSelector:
    matchLabels:
      release: prometheus
  portName: http-web
  replicas: 1
  resources:
    limits:
      cpu: "1"
      memory: 1000Mi
    requests:
      cpu: 500m
      memory: 750Mi
  retention: 3d
  routePrefix: /
  ruleNamespaceSelector: {}
  ruleSelector:
    matchLabels:
      release: prometheus
  rules:
    alert: {}
  scrapeInterval: 30s
  securityContext:
    fsGroup: 2000
    runAsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-kube-prometheus-prometheus
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector:
    matchLabels:
      release: prometheus
  version: v2.36.1

排查思路 #

故障核心大概率出在Alertmanager路由匹配环节，按以下顺序逐项排查：

• 确认Alertmanager侧是否收到告警
  打开Alertmanager Web UI的Alerts页面，检查是否存在test这条告警。如果Alertmanager侧无该告警，说明Prometheus到Alertmanager的推送链路异常；如果能看到该告警，问题锁定在Alertmanager的路由匹配、通知发送环节。
• 核对AlertmanagerConfig的路由匹配规则
  你当前创建的mailhog类型AlertmanagerConfig属于子路由配置，路由块中未配置matchers匹配规则，默认不会主动匹配告警。默认内置告警能正常发送，是因为内置规则标签刚好命中默认匹配逻辑，自定义告警未匹配到对应接收器就会被丢弃。
  可先在路由块中添加明确的匹配规则测试：

  route:
    matchers:
      - severity="warning"
    groupBy:
    - severity
    groupInterval: 30s
    groupWait: 30s
    receiver: mailhog
    repeatInterval: 1m
  

• 检查Alertmanager运行日志
  执行以下命令拉取Alertmanager日志，直接定位错误：

  kubectl logs -n prometheus prometheus-kube-prometheus-alertmanager-0
  

  重点搜索dropping alert、notify attempt failed关键字段，可直接看到告警被丢弃的原因、邮件发送失败的具体报错。
• 排除等待时间不足的误判
  你配置的groupWait为30s，告警触发后需要等待满分组等待时间才会发送通知，测试时至少等待30s以上再检查Mailhog收件箱，避免因为等待时长不足误判。
• 核对Alertmanager加载的配置
  在Alertmanager Web UI的Status页面，查看当前生效的完整配置，确认mailhog接收器、对应路由规则是否真的被加载，排除配置未同步生效的问题。

内容的提问来源于stack exchange，提问作者Paibok