如何为EKS集群中运行的Alertmanager应用alertmanager.yml配置

环境说明 #

我有一个运行在EKS上的Kubernetes集群，通过kube-prometheus仓库安装了Prometheus和Alertmanager，当前可通过localhost:9090访问Prometheus面板，集群中运行3个Alertmanager Pod，Pod列表如下：

$ kubectl get pod -A
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
kube-system   aws-node-669m7                         1/1     Running   0          29m
kube-system   aws-node-7rwhg                         1/1     Running   0          29m
kube-system   aws-node-f7tnf                         1/1     Running   0          39m
kube-system   cluster-autoscaler-78c99d7c59-6gbn7    1/1     Running   0          31m
kube-system   coredns-7dbf6bcd5b-5hgjx               1/1     Running   0          49m
kube-system   coredns-7dbf6bcd5b-ff5b4               1/1     Running   0          49m
kube-system   kube-proxy-jrwxh                       1/1     Running   0          29m
kube-system   kube-proxy-lwm9q                       1/1     Running   0          39m
kube-system   kube-proxy-xp9l9                       1/1     Running   0          29m
monitoring    alertmanager-main-0                    2/2     Running   0          35m
monitoring    alertmanager-main-1                    2/2     Running   0          35m
monitoring    alertmanager-main-2                    2/2     Running   0          35m
monitoring    blackbox-exporter-69f4d86566-d99bk     3/3     Running   0          36m
monitoring    grafana-79cd8d4b69-2k9np               1/1     Running   0          36m
monitoring    kube-state-metrics-56f8746666-q298b    3/3     Running   0          31m
monitoring    node-exporter-ndln7                    2/2     Running   0          29m
monitoring    node-exporter-psqng                    2/2     Running   0          29m
monitoring    node-exporter-z92vx                    2/2     Running   0          35m
monitoring    prometheus-adapter-77f56b865b-4sgv7    1/1     Running   0          31m
monitoring    prometheus-adapter-77f56b865b-f9tx6    1/1     Running   0          35m
monitoring    prometheus-k8s-0                       2/2     Running   0          35m
monitoring    prometheus-k8s-1                       2/2     Running   0          35m
monitoring    prometheus-operator-54f6687b97-5mdbb   2/2     Running   0          35m

我需要配置告警规则：当集群节点数超过4时，向我的Gmail账户发送邮件。已编写示例alertmanager.yml配置文件，但本地没有alertmanager命令，不知道如何将配置应用到集群中的Alertmanager Pod。

我的示例配置：

global:
  resolve_timeout: 5m
  smtp_from: 'prometheus.test@example.com'
  smtp_smarthost: 'localhost:25'
route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'web.hook'
  routes:
  - receiver: 'mailtest'
    match:
      test_type: SampleAlert
receivers:
- name: 'mailtest'
  email_configs:
    - to: 'root@localhost.localdomain'
      require_tls: false
- name: 'web.hook'
  webhook_configs:
  - url: 'https://127.0.0.1:5001/'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

解决方案 #

kube-prometheus通过Prometheus Operator管理Alertmanager，配置存储在Kubernetes Secret中，无需本地安装alertmanager命令，按以下步骤操作：

1. 调整Alertmanager配置（适配Gmail） #

先修改你的alertmanager.yml，配置正确的Gmail SMTP参数（Gmail需开启两步验证并生成应用专用密码）：

global:
  resolve_timeout: 5m
  smtp_from: '你的Gmail邮箱@gmail.com'
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_auth_username: '你的Gmail邮箱@gmail.com'
  smtp_auth_password: '你的Gmail应用专用密码'
  smtp_require_tls: true
route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'web.hook'
  routes:
  - receiver: 'mailtest'
    match:
      test_type: SampleAlert
receivers:
- name: 'mailtest'
  email_configs:
    - to: '你的目标邮箱@gmail.com'
      require_tls: true
- name: 'web.hook'
  webhook_configs:
  - url: 'https://127.0.0.1:5001/'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

2. 更新Alertmanager配置到集群Secret #

kube-prometheus默认使用alertmanager-main Secret存储配置，执行以下命令更新：

# 用本地配置文件覆盖集群中的Secret
kubectl create secret generic alertmanager-main --from-file=alertmanager.yml -n monitoring --dry-run=client -o yaml | kubectl apply -f -

3. 重启Alertmanager Pod加载新配置 #

Secret更新后Pod不会自动重启，需触发StatefulSet滚动更新：

kubectl rollout restart statefulset alertmanager-main -n monitoring

4. 配置Prometheus节点数告警规则 #

告警规则需配置在Prometheus中，创建node-count-alert.yaml文件：

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: node-count-alert
  namespace: monitoring
  labels:
    prometheus: k8s
    role: alert-rules
spec:
  groups:
  - name: node-count
    rules:
    - alert: NodeCountExceeds4
      expr: count(kube_node_info{job="kube-state-metrics"}) > 4
      for: 1m
      labels:
        severity: warning
        test_type: SampleAlert # 和Alertmanager配置中的match规则对应
      annotations:
        summary: "集群节点数超过4个"
        description: "当前节点数: {{ $value }}"

应用规则到集群：

kubectl apply -f node-count-alert.yaml -n monitoring

内容的提问来源于stack exchange，提问作者whitebear