如何通过自定义ConfigMap为Prometheus加载自定义告警规则？

步骤1：创建包含告警规则的ConfigMap #

先编写自定义告警规则文件（比如soti-alerts-rules.yaml）：

groups:
- name: soti-custom-alerts
  rules:
  - alert: HighPodCPUUsage
    expr: sum(rate(container_cpu_usage_seconds_total{namespace!="kube-system"}[5m])) by (pod) > 0.8
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Pod {{ $labels.pod }} CPU使用率过高"
      description: "Pod {{ $labels.pod }} 最近5分钟CPU使用率超过80%"

执行命令创建ConfigMap：

kubectl create configmap soti-alerts -n monitoring --from-file=soti-alerts-rules.yaml

步骤2：修正Helm的extra_values.yaml配置 #

之前的配置仅完成了ConfigMap挂载，未让Prometheus加载规则目录，也未配置reloader监控新目录。更新extra_values.yaml如下：

prometheus:
  prometheusSpec:
    # 关闭默认规则选择器的Helm值依赖，确保自定义规则能被加载
    ruleSelectorNilUsesHelmValues: false
    # 让Prometheus同时加载默认规则目录和自定义规则目录下的所有YAML文件
    ruleFiles:
      - /etc/prometheus/rules/prometheus-monitoring-kube-prometheus-prometheus-rulefiles-0/*.yaml
      - /etc/prometheus/rules/soti-alerts/*.yaml
    # 将自定义ConfigMap挂载到Prometheus Pod
    additionalMounts:
      - name: soti-alerts
        mountPath: /etc/prometheus/rules/soti-alerts
        readOnly: true

configmapReload:
  prometheus:
    enabled: true
    # 让配置重载器同时监控默认规则目录和自定义规则目录
    extraArgs:
      - --watched-dir=/etc/prometheus/rules/prometheus-monitoring-kube-prometheus-prometheus-rulefiles-0
      - --watched-dir=/etc/prometheus/rules/soti-alerts
    # 将自定义ConfigMap挂载到重载器容器
    extraConfigmapMounts:
       - name: soti-alerts
         mountPath: /etc/prometheus/rules/soti-alerts
         configMap: soti-alerts
         readOnly: true

步骤3：重新应用Helm模板 #

执行原有Helm模板命令，重新生成并应用配置：

helm template monitoring \
  --namespace monitoring \
  prometheus-community/kube-prometheus-stack --version 41.5.1 \
  -f extra_values.yaml \
  --set grafana.smtp.existingSecret=grafana-smtp \
  --set grafana.adminPassword=${GRAFANA_ADMIN_PASSWORD} \
  --set grafana.ingress.hosts=\{grafana.${ROOT_DNS}\} \
  --set prometheus.ingress.hosts=\{prometheus.${ROOT_DNS}\} \
  | kubectl -n monitoring apply -f-

步骤4：验证配置 #

1. 查看Prometheus Pod的init容器参数，确认--watched-dir包含两个目录：

   kubectl describe pod <prometheus-pod-name> -n monitoring | grep -A 10 "init-config-reloader"
   

2. 访问Prometheus UI的Alerts页面，检查自定义告警规则是否成功加载。
3. 更新规则时，直接编辑ConfigMap即可，配置重载器会自动检测变化并重新加载规则：

   kubectl edit configmap soti-alerts -n monitoring
   

内容的提问来源于stack exchange，提问作者Victor Rodriguez