Kibana Watcher如何按错误匹配条件分群组发送告警邮件

现有基础 #

• 已完成Kibana Watcher错误分类统计配置，分类效果如下：
  
• 已实现通用错误匹配逻辑：当错误key包含指定字符串时，可向固定邮箱发送告警邮件，当前需新增按错误类型匹配对应收件组的路由能力。

路由规则 #

• 错误key匹配Response status code does not indicate success Service Unavailable时，向群组1发信，收件人列表：user1@gmail.com、user2@gmail.com、user3@gmail.com
• 错误key匹配Response status code does not indicate success Gateway时，向群组2发信，收件人列表：user4@gmail.com、user5@gmail.com、user6@gmail.com

原有配置 #

"actions": {
"send_email": {
  "throttle_period_in_millis": 300000,
  "condition": {
    "script": {
      "source": " def status = false; for(int i=0; i<ctx.payload.failure_request.aggregations.categories.buckets.length;i++) {if(ctx.payload.failure_request.aggregations.categories.buckets[i].key.contains('Response status code does not indicate success')) {status = true}} return status ",
      "lang": "painless"
    }
  },
  "email": {
    "profile": "standard",
    "to": [
      "avinash.singh1@spglobal.com"
    ],
    "subject": "{{ctx.metadata.email_subject}}",
    "body": {
      "html": "Error Found: <ul> {{ctx.payload.aggregations.categories.buckets.length}}"
                   }
                 }
              }
            }

实现方案 #

Painless支持switch-case分支逻辑，但针对字符串包含匹配的路由场景，用映射表预处理的方式扩展性更强，维护成本更低，以下提供两种可直接使用的实现方式：

方案1：单动作动态计算收件人（推荐） #

通过transform脚本提前遍历聚合结果，匹配错误类型后自动汇总去重收件人列表，邮件动作直接读取预处理后的收件人即可，无需维护多个重复动作：

"transform": {
  "script": {
    "source": """
      // 初始化去重收件人集合
      def recipients = new HashSet();
      def buckets = ctx.payload.failure_request.aggregations.categories.buckets;
      // 维护错误类型与收件组的映射关系，后续新增规则直接在此处添加即可
      def errMailMap = [
        "Response status code does not indicate success Service Unavailable": ["user1@gmail.com", "user2@gmail.com", "user3@gmail.com"],
        "Response status code does not indicate success Gateway": ["user4@gmail.com", "user5@gmail.com", "user6@gmail.com"]
      ];
      // 遍历错误桶匹配对应收件人
      for (def bucket : buckets) {
        for (def errKey : errMailMap.keySet()) {
          if (bucket.key.contains(errKey)) {
            recipients.addAll(errMailMap[errKey]);
          }
        }
      }
      ctx.targetRecipients = recipients;
      return ctx;
    """,
    "lang": "painless"
  }
},
"actions": {
  "send_alert_mail": {
    "throttle_period_in_millis": 300000,
    "condition": {
      "script": {
        "source": "return ctx.targetRecipients.size() > 0",
        "lang": "painless"
      }
    },
    "email": {
      "profile": "standard",
      "to": "{{#ctx.targetRecipients}}{{this}},{{/ctx.targetRecipients}}",
      "subject": "{{ctx.metadata.email_subject}}",
      "body": {
        "html": "检测到如下错误：<ul>{{#ctx.payload.failure_request.aggregations.categories.buckets}}<li>错误类型：{{key}}，触发次数：{{doc_count}}</li>{{/ctx.payload.failure_request.aggregations.categories.buckets}}</ul>"
      }
    }
  }
}

该方案优势：后续新增错误路由规则仅需修改errMailMap映射表，自动实现收件人去重，避免同一用户收到重复告警，同时修正了原有配置中聚合路径写错的问题（原代码body中取的是ctx.payload.aggregations，实际路径应为ctx.payload.failure_request.aggregations）。

方案2：多动作独立匹配 #

如果需要给不同收件组发送差异化的邮件内容/主题，可以拆分多个独立的邮件动作，每个动作单独配置匹配条件：

"actions": {
  "send_mail_group1": {
    "throttle_period_in_millis": 300000,
    "condition": {
      "script": {
        "source": """
          def match = false;
          def buckets = ctx.payload.failure_request.aggregations.categories.buckets;
          for (def bucket : buckets) {
            if (bucket.key.contains("Response status code does not indicate success Service Unavailable")) {
              match = true;
              break;
            }
          }
          return match;
        """,
        "lang": "painless"
      }
    },
    "email": {
      "profile": "standard",
      "to": ["user1@gmail.com", "user2@gmail.com", "user3@gmail.com"],
      "subject": "{{ctx.metadata.email_subject}} - Service Unavailable告警",
      "body": {
        "html": "检测到Service Unavailable类型错误，请及时排查处理"
      }
    }
  },
  "send_mail_group2": {
    "throttle_period_in_millis": 300000,
    "condition": {
      "script": {
        "source": """
          def match = false;
          def buckets = ctx.payload.failure_request.aggregations.categories.buckets;
          for (def bucket : buckets) {
            if (bucket.key.contains("Response status code does not indicate success Gateway")) {
              match = true;
              break;
            }
          }
          return match;
        """,
        "lang": "painless"
      }
    },
    "email": {
      "profile": "standard",
      "to": ["user4@gmail.com", "user5@gmail.com", "user6@gmail.com"],
      "subject": "{{ctx.metadata.email_subject}} - Gateway错误告警",
      "body": {
        "html": "检测到Gateway类型错误，请及时排查处理"
      }
    }
  }
}

注意：如果使用多动作方案，建议给不同动作设置独立的节流周期标识，避免同类型错误短时间内重复发信。

内容的提问来源于stack exchange，提问作者Avinash Singh