FluentBit传输Docker容器应用日志到ELK：去除附加前缀行问题

问题描述 #

需将以下Mule生成的JSON日志解析并发送至ELK：

[2023-12-05 11:24:35.281] INFO  JsonLogger [[MuleRuntime].uber.8614: [rtf-eks-vali-dev].post:\notification:api-config.BLOCKING @32b79a38] [event: de6ad1b0-9360-11ee-9395-8e6b4b575608]: {
  "correlationId" : "de6ad1b0-9360-11ee-9395-8e6b4b575608",
  "message" : "This is O/P from JSON Logger",
  "tracePoint" : "EVENT",
  "priority" : "INFO",
  "elapsed" : 0,
  "locationInfo" : {
    "lineInFile" : "138",
    "component" : "json-logger:logger",
    "fileName" : "api-routes.xml",
    "rootContainer" : "post:\\notification:api-config"
  },
  "timestamp" : "2023-12-05T11:24:35.281Z",
  "applicationName" : "cn-memory-check-sys",
  "applicationVersion" : "1.0.0-SNAPSHOT",
  "environment" : "dev",
  "threadName" : "[MuleRuntime].uber.8614: [rtf-eks-vali-dev].post:\\notification:api-config.BLOCKING @32b79a38",
  "metadata" : {
    "User-Agent" : "PostmanRuntime/7.35.0",
    "Host" : "dev-cn1-apis.com.cn",
    "Content-Length" : "227",
    "Content-Type" : "text/plain"
  }
}

但每条JSON元素前均被附加类似2022-07-25T06:43:17.20958947Z stdout F的Docker前缀行，导致JSON结构破坏，无法正常解析。

解决方案 #

方案1：使用Filebeat处理（轻量高效） #

通过Filebeat的多行合并、字段拆分和JSON解码功能，直接在采集阶段处理日志：

filebeat.inputs:
- type: container
  paths:
    - /var/lib/docker/containers/*/*.log
  # 合并多行JSON日志
  multiline.pattern: '^\{'
  multiline.negate: true
  multiline.match: after
  multiline.flush_pattern: '^\}'

processors:
  # 拆分Docker前缀与实际日志内容
  - dissect:
      tokenizer: "%{timestamp} %{stream} %{flag} %{message}"
      field: "message"
      target_prefix: "docker"
  # 移除无用的Docker前缀字段
  - drop_fields:
      fields: ["docker.timestamp", "docker.stream", "docker.flag"]
  # 解码JSON内容到顶层字段
  - decode_json_fields:
      fields: ["message"]
      target: ""
      overwrite_keys: true

output.elasticsearch:
  hosts: ["http://your-es-host:9200"]
  index: "mule-app-logs-%{+YYYY.MM.dd}"

方案2：使用Logstash过滤处理 #

若已通过Logstash接收日志，可通过多行合并、Grok匹配和JSON解码完成处理：

input {
  beats {
    port => 5044
  }
}

filter {
  # 合并多行JSON为单条日志
  multiline {
    pattern => "^\{"
    negate => true
    what => "previous"
    auto_flush_interval => 5
  }

  # 匹配并剥离Docker前缀
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:docker_timestamp} %{WORD:docker_stream} %{WORD:docker_flag} %{GREEDYDATA:raw_message}" }
    remove_field => ["message", "docker_timestamp", "docker_stream", "docker_flag"]
  }

  # 解析JSON内容到顶层字段
  json {
    source => "raw_message"
    target => ""
    remove_field => ["raw_message"]
  }
}

output {
  elasticsearch {
    hosts => ["http://your-es-host:9200"]
    index => "mule-app-logs-%{+YYYY.MM.dd}"
  }
}

方案3：调整Docker日志配置（源头优化） #

修改Docker daemon配置文件/etc/docker/daemon.json，优化日志输出格式，减少后续处理成本：

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "tag": "{{.Name}}"
  }
}

修改后重启Docker服务：

systemctl restart docker

此配置会让Docker将日志以JSON格式存储，Filebeat采集时可直接读取log字段中的原始日志内容，避免前缀干扰。

内容的提问来源于stack exchange，提问作者Pratik Agrawal