Fluent Bit grep过滤器未按预期工作，求正确配置方法

问题 #

尝试按日志级别过滤Fluent Bit日志，但grep过滤器未按预期工作。

样本日志 #

[1] sample: [1678822837.893110278, {"log"=>"{\"Timestamp\":\"2023-03-15T03:40:37.8929481+08:00\",\"Level\":\"Verbose\",\"MessageTemplate\":\"This is Trace\",\"Properties\":{\"ActionId\":\"2c8e96ec-7ea7-4bf5-bd8f-e692fecae4a7\",\"RequestId\":\"0HMP4O98PRMG2:00000001\",\"RequestPath\":\"/SampleLogging\",\"SpanId\":\"|e8ff0813-405598a6b4fe004d.\",\"TraceId\":\"e8ff0813-405598a6b4fe004d\",\"ParentId\":\"\",\"ConnectionId\":\"0HMP4O98PRMG2\"}}"}]
[2] sample: [1678822837.893486359, {"log"=>"{\"Timestamp\":\"2023-03-15T03:40:37.8933146+08:00\",\"Level\":\"Debug\",\"MessageTemplate\":\"This is Debug\",\"Properties\":{\"ActionId\":\"2c8e96ec-7ea7-4bf5-bd8f-e692fecae4a7\",\"RequestId\":\"0HMP4O98PRMG2:00000001\",\"RequestPath\":\"/SampleLogging\",\"SpanId\":\"|e8ff0813-405598a6b4fe004d.\",\"TraceId\":\"e8ff0813-405598a6b4fe004d\",\"ParentId\":\"\",\"ConnectionId\":\"0HMP4O98PRMG2\"}}"}]
[3] sample: [1678822837.893674630, {"log"=>"{\"Timestamp\":\"2023-03-15T03:40:37.8935494+08:00\",\"Level\":\"Information\",\"MessageTemplate\":\"This is Information\",\"Properties\":{\"ActionId\":\"2c8e96ec-7ea7-4bf5-bd8f-e692fecae4a7\",\"RequestId\":\"0HMP4O98PRMG2:00000001\",\"RequestPath\":\"/SampleLogging\",\"SpanId\":\"|e8ff0813-405598a6b4fe004d.\",\"TraceId\":\"e8ff0813-405598a6b4fe004d\",\"ParentId\":\"\",\"ConnectionId\":\"0HMP4O98PRMG2\"}}"}]
[4] sample: [1678822837.893866881, {"log"=>"{\"Timestamp\":\"2023-03-15T03:40:37.8937288+08:00\",\"Level\":\"Warning\",\"MessageTemplate\":\"This is Warning\",\"Properties\":{\"ActionId\":\"2c8e96ec-7ea7-4bf5-bd8f-e692fecae4a7\",\"RequestId\":\"0HMP4O98PRMG2:00000001\",\"RequestPath\":\"/SampleLogging\",\"SpanId\":\"|e8ff0813-405598a6b4fe004d.\",\"TraceId\":\"e8ff0813-405598a6b4fe004d\",\"ParentId\":\"\",\"ConnectionId\":\"0HMP4O98PRMG2\"}}"}]
[5] sample: [1678822837.894049382, {"log"=>"{\"Timestamp\":\"2023-03-15T03:40:37.8939219+08:00\",\"Level\":\"Error\",\"MessageTemplate\":\"This is Error\",\"Properties\":{\"ActionId\":\"2c8e96ec-7ea7-4bf5-bd8f-e692fecae4a7\",\"RequestId\":\"0HMP4O98PRMG2:00000001\",\"RequestPath\":\"/SampleLogging\",\"SpanId\":\"|e8ff0813-405598a6b4fe004d.\",\"TraceId\":\"e8ff0813-405598a6b4fe004d\",\"ParentId\":\"\",\"ConnectionId\":\"0HMP4O98PRMG2\"}}"}]
[6] sample: [1678822837.894352233, {"log"=>"{\"Timestamp\":\"2023-03-15T03:40:37.8941080+08:00\",\"Level\":\"Fatal\",\"MessageTemplate\":\"This is Fatal\",\"Properties\":{\"ActionId\":\"2c8e96ec-7ea7-4bf5-bd8f-e692fecae4a7\",\"RequestId\":\"0HMP4O98PRMG2:00000001\",\"RequestPath\":\"/SampleLogging\",\"SpanId\":\"|e8ff0813-405598a6b4fe004d.\",\"TraceId\":\"e8ff0813-405598a6b4fe004d\",\"ParentId\":\"\",\"ConnectionId\":\"0HMP4O98PRMG2\"}}"}]

尝试过的命令 #

1. 筛选Warning/Error/Fatal级别，但无输出：

bin/fluent-bit -i tail \
  -p 'path=/home/user/app/logs/*.*' \
  -p 'tag=sample' \
  -F grep \
  -p 'regex=Level Warning|Error|Fatal' \
  -m '*' \
  -o stdout

2. 排除Information级别，但该级别日志仍能通过：

bin/fluent-bit -i tail \
  -p 'path=/home/user/app/logs/*.*' \
  -p 'tag=sample' \
  -F grep \
  -p 'exclude=Level Information' \
  -m '*' \
  -o stdout

解决方案 #

问题核心在于日志结构嵌套层级和grep过滤器的匹配逻辑：

1. Level字段并非顶级字段，而是嵌套在log字段的JSON字符串中，grep默认无法直接解析嵌套JSON，需先解析结构化字段。
2. 原命令的正则匹配格式错误，未遵循"字段名+精确正则"的规则。

正确配置示例 #

筛选Warning/Error/Fatal级别

bin/fluent-bit -i tail \
  -p 'path=/home/user/app/logs/*.*' \
  -p 'tag=sample' \
  -F parser \
  -p 'name=json' \
  -p 'key_name=log' \
  -m '*' \
  -F grep \
  -p 'regex=Level ^(Warning|Error|Fatal)$' \
  -m '*' \
  -o stdout

排除Information级别

bin/fluent-bit -i tail \
  -p 'path=/home/user/app/logs/*.*' \
  -p 'tag=sample' \
  -F parser \
  -p 'name=json' \
  -p 'key_name=log' \
  -m '*' \
  -F grep \
  -p 'exclude=Level ^Information$' \
  -m '*' \
  -o stdout

关键说明 #

• parser过滤器的作用：将log字段中的JSON字符串解析为结构化字段，使Level成为可直接匹配的顶级字段。
• 正则匹配规则：
  • 使用^(Warning|Error|Fatal)$确保完全匹配指定级别，避免部分匹配（比如避免误匹配包含Warning的其他字符串）。
  • ^和$分别表示字符串的开头和结尾，保证精确匹配日志级别。

内容的提问来源于stack exchange，提问作者Kelvin Go