AlertmanagerConfig匹配规则路由异常问题求助

问题根源 #

你当前的问题是多AlertmanagerConfig实例的路由规则合并冲突。每个业务线的配置都定义了完整的路由树（包括默认接收者、兜底null路由），当Alertmanager将这些配置合并时，多个重叠的路由规则会导致告警被随机匹配到不同的接收者，最终出现频道跳转的异常。

解决步骤 #

1. 重构配置结构，拆分路由职责 #

• 单独创建根路由配置，负责全局路由的基础设置（分组、间隔、兜底接收）。
• 业务线的AlertmanagerConfig仅定义自身标签匹配的路由分支，不再包含完整路由树。

根路由配置示例（root-sit-config）

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: root-sit-config
  namespace: monitoring
  labels:
    alertmanagerConfig: "sit-config"
spec:
  route:
    groupBy: ['alertname', 'namespace']
    groupWait: '30s'
    groupInterval: '5m'
    repeatInterval: '15m'
    # 全局兜底接收者，避免未匹配的告警扩散
    receiver: 'null'
  receivers:
    - name: 'null'
      webhookConfigs: []

改造后的abc-api配置

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: abc-api
  namespace: abc-sit
  labels:
    alertmanagerConfig: "sit-config"
spec:
  route:
    routes:
      - matchers:
          - name: 'alert_source'
            value: 'abc-api'
            matchType: =~
        receiver: 'abc-sit-default'
  receivers:
    - name: 'abc-sit-default'
      slackConfigs:
        - sendResolved: true
          apiURL:
            name: "abc-api-prom"
            key: defaultWebhook
          channel: "#abc-api-nonprod-alerts"
          title: |-
            [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.deployment }} in {{ .CommonLabels.environment }}
          text: |-
            *Alert:* {{ .CommonAnnotations.summary }}

改造后的xyz配置

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: xyz
  namespace: xyz-sit
  labels:
    alertmanagerConfig: "sit-config"
spec:
  route:
    routes:
      - matchers:
          - name: severity
            value: critical
            matchType: =~
          - name: alert_source
            value: xyz
            matchType: =~
        receiver: 'xyz-sit-critical'
      - matchers:
          - name: severity
            value: warning
            matchType: =~
          - name: alert_source
            value: xyz
            matchType: =~
        receiver: 'xyz-sit-warning'
      - matchers:
          - name: alert_source
            value: xyz
            matchType: =~
        receiver: 'xyz-sit-default'
  receivers:
    - name: 'xyz-sit-default'
      slackConfigs:
        - sendResolved: true
          apiURL:
            name: "xyz-prom"
            key: defaultWebhook
          channel: "#alert-testing"
          title: |-
            [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.deployment }} in {{ .CommonLabels.environment }}
          text: |-
            *Alert:* {{ .CommonAnnotations.summary }}
    - name: 'xyz-sit-critical'
      slackConfigs:
        - sendResolved: true
          apiURL:
            name: "xyz-prom"
            key: critical
          channel: "#alert-testing"
          title: |-
            [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.deployment }} in {{ .CommonLabels.environment }}
          text: |-
            *Alert:* {{ .CommonAnnotations.summary }}
    - name: 'xyz-sit-warning'
      slackConfigs:
        - sendResolved: true
          apiURL:
            name: "xyz-prom"
            key: warning
          channel: "#alert-testing"
          title: |-
            [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.deployment }} in {{ .CommonLabels.environment }}
          text: |-
            *Alert:* {{ .CommonAnnotations.summary }}

改造后的dfg配置

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: dfg
  namespace: dfg-sit
  labels:
    alertmanagerConfig: "sit-config"
spec:
  route:
    routes:
      - matchers:
          - name: severity
            value: critical
            matchType: =~
          - name: alert_source
            value: dfg
            matchType: =~
        receiver: 'dfg-sit-critical'
      - matchers:
          - name: severity
            value: warning
            matchType: =~
          - name: alert_source
            value: dfg
            matchType: =~
        receiver: 'dfg-sit-warning'
      - matchers:
          - name: alert_source
            value: dfg
            matchType: =~
        receiver: 'dfg-sit-default'
  receivers:
    - name: 'dfg-sit-default'
      slackConfigs:
        - sendResolved: true
          apiURL:
            name: "dfg-prom"
            key: defaultWebhook
          channel: "#dfg-nonprod-alerts"
          title: |-
            [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.deployment }} in {{ .CommonLabels.environment }}
          text: |-
            *Alert:* {{ .CommonAnnotations.summary }}
    - name: 'dfg-sit-critical'
      slackConfigs:
        - sendResolved: true
          apiURL:
            name: "dfg-prom"
            key: critical
          channel: "#dfg-nonprod-alerts"
          title: |-
            [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.deployment }} in {{ .CommonLabels.environment }}
          text: |-
            *Alert:* {{ .CommonAnnotations.summary }}
    - name: 'dfg-sit-warning'
      slackConfigs:
        - sendResolved: true
          apiURL:
            name: "dfg-prom"
            key: warning
          channel: "#dfg-nonprod-alerts"
          title: |-
            [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.deployment }} in {{ .CommonLabels.environment }}
          text: |-
            *Alert:* {{ .CommonAnnotations.summary }}

2. 移除冗余的空值匹配规则 #

所有业务线配置中alert_source: ""的匹配规则全部删除，统一由根配置的null接收者处理未匹配的告警，避免多规则冲突。

3. 验证合并后的配置 #

• 查看集群中合并后的Alertmanager配置：

  kubectl get alertmanagerconfigs -l alertmanagerConfig=sit-config -o yaml
  

• 进入Alertmanager Pod内部，使用工具查看最终生效的路由规则：

  amtool config show
  

关键原理 #

Alertmanager会将所有带有相同alertmanagerConfig标签的AlertmanagerConfig实例合并为一个完整的路由树。如果多个实例都定义了根路由的默认接收者，合并后会导致规则优先级混乱，告警会被最先匹配到的任意规则处理，从而出现随机跳转的情况。通过拆分根路由和业务分支，确保路由规则清晰无重叠，就能解决这个问题。

内容的提问来源于stack exchange，提问作者Icarus