如何在Fluentbit中提取AlertManager告警的commonAnnotations.description字段

需求：将AlertManager发送至Fluentbit的告警里，提取嵌套在commonAnnotations下的description字段，输出到Syslog服务器。

AlertManager告警示例 #

{
  "receiver": "fluentbit-webhook",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "KubeJobFailed",
        "condition": "true",
        "container": "kube-state-metrics",
        "endpoint": "http",
        "instance": "10.42.6.188:8080",
        "job": "kube-state-metrics",
        "job_name": "helm-install-aws-ebs-csi-driver",
        "namespace": "kube-system",
        "pod": "prometheus-operator-kube-state-metrics-59c8dc555f-l7dlv",
        "prometheus": "monitoring/prometheus-operator-kube-p-prometheus",
        "service": "prometheus-operator-kube-state-metrics",
        "severity": "warning"
      },
      "annotations": {
        "description": "Job kube-system/helm-install-aws-ebs-csi-driver failed to complete. Removing failed job after investigation should clear this alert.",
        "runbook_url": "https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubejobfailed",
        "summary": "Job failed to complete."
      },
      "startsAt": "2023-10-05T09:21:25.327Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://prometheus.monitoring.core.oxygen.example.com/graph?g0.expr=kube_job_failed%7Bjob%3D%22kube-state-metrics%22%2Cnamespace%3D~%22.%2A%22%7D+%3E+0&g0.tab=1",
      "fingerprint": "1a5cd56a32bc18c2"
    }
  ],
  "groupLabels": {
    "namespace": "kube-system"
  },
  "commonLabels": {
    "alertname": "KubeJobFailed",
    "condition": "true",
    "container": "kube-state-metrics",
    "endpoint": "http",
    "instance": "10.42.6.188:8080",
    "job": "kube-state-metrics",
    "job_name": "helm-install-aws-ebs-csi-driver",
    "namespace": "kube-system",
    "pod": "prometheus-operator-kube-state-metrics-59c8dc555f-l7dlv",
    "prometheus": "monitoring/prometheus-operator-kube-p-prometheus",
    "service": "prometheus-operator-kube-state-metrics",
    "severity": "warning"
  },
  "commonAnnotations": {
    "description": "Job kube-system/helm-install-aws-ebs-csi-driver failed to complete. Removing failed job after investigation should clear this alert.",
    "runbook_url": "https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubejobfailed",
    "summary": "Job failed to complete."
  },
  "externalURL": "http://alertmanager.monitoring.core.oxygen.example.com",
  "version": "4",
  "groupKey": "{/{severity=\"warning\"}:{namespace=\"kube-system\"}}",
  "truncatedAlerts": 0
}

现有Fluentbit Syslog输出配置 #

[OUTPUT]
    Name syslog
    Match syslog.*
    Host bastion.dev.oxyproj.net
    Port 514
    Retry_Limit false
    Mode tcp
    Syslog_Format rfc5424
    Syslog_MaxSize 65536
    Syslog_Hostname_Key hostname
    Syslog_Appname_Key appname
    Syslog_Procid_Key procid
    Syslog_Msgid_Key msgid
    Syslog_SD_Key uls@0
    Syslog_Message_Key msg

解决方案 #

Fluentbit的Syslog输出插件无法直接引用嵌套JSON字段，需先通过过滤器将目标字段提升至日志顶层，再配置输出引用该字段。

方法1：使用nest过滤器批量提取嵌套字段 #

该方法会将commonAnnotations下的所有字段都提升到顶层，并添加指定前缀，适合需要提取多个嵌套字段的场景：

[FILTER]
    Name nest
    Match alertmanager.*  # 替换为接收AlertManager告警的输入标签
    Operation lift
    Nested_under commonAnnotations
    Add_prefix alert_

配置生效后，commonAnnotations.description会被提取为顶层字段alert_description。

方法2：使用lua过滤器精准提取单个字段 #

如果只需要提取description字段，用Lua过滤器更灵活：

1. 创建Lua脚本文件（比如/etc/fluentbit/extract_alert_description.lua）：

function extract_alert_description(tag, timestamp, record)
    if record.commonAnnotations and record.commonAnnotations.description then
        record.alert_description = record.commonAnnotations.description
    end
    return 2, timestamp, record
end

2. 在Fluentbit配置中添加过滤器引用该脚本：

[FILTER]
    Name lua
    Match alertmanager.*  # 替换为对应输入标签
    call extract_alert_description
    script /etc/fluentbit/extract_alert_description.lua

修改Syslog输出配置 #

将输出配置中的Syslog_Message_Key替换为提取后的顶层字段名：

[OUTPUT]
    Name syslog
    Match syslog.*  # 确保与过滤器的Match规则匹配
    Host bastion.dev.oxyproj.net
    Port 514
    Retry_Limit false
    Mode tcp
    Syslog_Format rfc5424
    Syslog_MaxSize 65536
    Syslog_Hostname_Key hostname
    Syslog_Appname_Key appname
    Syslog_Procid_Key procid
    Syslog_Msgid_Key msgid
    Syslog_SD_Key uls@0
    Syslog_Message_Key alert_description  # 引用提取后的顶层字段

验证 #

重启Fluentbit服务，触发AlertManager告警后，检查Syslog服务器是否收到包含目标description内容的日志条目。

内容的提问来源于stack exchange，提问作者Raz Maabari