Filebeat+Logstash推送Docker容器日志至ClickHouse失败问题排查

问题描述 #

配置Filebeat与Logstash，尝试将Docker容器日志推送至ClickHouse但失败，怀疑ClickHouse日志表与Logstash处理的日志结构不匹配。使用以下Docker命令运行Filebeat：

docker run --rm -it --name=filebeat --user=root \
--volume="/var/log/:/var/log/" \
--volume="/var/lib/docker/containers/:/var/lib/docker/containers/" \
--volume="$(pwd)/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml" \
docker.elastic.co/beats/filebeat:5.6.3 filebeat -e -strict.perms=false

filebeat.yml

filebeat.prospectors:
  - input_type: log
    paths:
      - /var/log/*.log
      - /var/lib/docker/containers/*.log   
    exclude_files: ['.gz$']
output.logstash:
  hosts: ['localhost:5044']

logstash.conf

input {
  beats {
    port => 5044
  }
}

filter {
      grok {
        match => [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}"]
      }

      mutate {
        convert => ["response", "integer"]
        convert => ["bytes", "integer"]
        convert => ["responsetime", "float"]
        remove_field => ["@version", "host", "message", "beat", "offset", "type", "tags", "input_type", "source"]
      }

      date {
        match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
        remove_field => [ "timestamp", "@timestamp" ]
        target => [ "logdatetime" ]
      }

      ruby {
        code => "tstamp = event.get('logdatetime').to_i
                 event.set('logdatetime', Time.at(tstamp).strftime('%Y-%m-%d %H:%M:%S'))
                 event.set('logdate', Time.at(tstamp).strftime('%Y-%m-%d'))"
      }

      useragent {
        source => "agent"
      }

     prune {
          interpolate => true
          whitelist_names => ["^logdate$" ,"^logdatetime$" ,"^request$" ,"^agent$" ,"^os$" ,"^minor$" ,"^auth$" ,"^ident$" ,"^verb$" ,"^patch$" ,"^referrer$" ,"^major$" ,"^build$" ,"^response$","^bytes$","^clientip$" ,"^name$" ,"^os_name$" ,"^httpversion$" ,"^device$" ]
        }
}
output {
    clickhouse {
      http_hosts => ["http://localhost:8123"]
      table => "nginx_access"
      request_tolerance => 1
      flush_size => 1000
      pool_max => 1000
    }
}

filebeat日志

2023/02/09 09:45:22.247315 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2023/02/09 09:45:22.247331 prospector.go:124: INFO Starting prospector of type: log; id: 7589403446011535719 
2023/02/09 09:45:22.247356 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2023/02/09 09:45:22.247929 log.go:91: INFO Harvester started for file: /var/log/falcond.log
2023/02/09 09:45:22.247982 log.go:91: INFO Harvester started for file: /var/log/yum.log
2023/02/09 09:45:22.248088 log.go:91: INFO Harvester started for file: /var/log/falconctl.log
2023/02/09 09:45:22.248622 log.go:91: INFO Harvester started for file: /var/log/boot.log
2023/02/09 09:45:22.248665 log.go:91: INFO Harvester started for file: /var/log/falcon-sensor.log
2023/02/09 09:45:52.239459 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=5 filebeat.harvester.running=5 filebeat.harvester.started=5 libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=24 libbeat.logstash.publish.write_bytes=3764 libbeat.logstash.published_and_acked_events=77 libbeat.publisher.published_events=77 publish.events=82 registrar.states.current=5 registrar.states.update=82 registrar.writes=2
2023/02/09 09:46:22.239403 metrics.go:34: INFO No non-zero metrics in the last 30s
2023/02/09 09:46:52.239518 metrics.go:34: INFO No non-zero metrics in the last 30s
2023/02/09 09:47:22.239436 metrics.go:34: INFO No non-zero metrics in the last 30s

排查与解决建议 #

1. 验证ClickHouse表结构匹配性
   检查nginx_access表的字段名、类型是否与Logstash prune阶段的白名单字段完全对应：

   • 例如logdate需为DATE或String类型（格式YYYY-MM-DD），logdatetime为DATETIME或String类型，response为Int类型，bytes为Int类型，responsetime为Float类型等。
   • 若表字段缺失或类型不匹配，直接导致写入失败。

2. 检查Logstash日志处理有效性

   • 当前grok规则仅适配Nginx的Combined日志格式，但Docker容器日志多为JSON结构（包含log、stream、time等字段），直接匹配会失败，导致后续字段缺失。
   • 临时修改Logstash输出，添加stdout { codec => rubydebug }，查看处理后的事件结构，确认字段是否完整。

3. 修正Docker日志收集方式

   • Filebeat 5.6.3支持Docker日志专用输入，可替换input_type: log为input_type: docker，并配置docker.containers.ids: '*'直接采集容器日志（无需手动挂载容器日志目录）；
   • 若坚持使用log输入类型，需在Logstash filter中添加json { source => "message" }解析Docker的JSON日志格式。

4. 查看Logstash运行日志
   Filebeat日志显示事件已成功发送并收到ACK，但写入ClickHouse的错误只会出现在Logstash日志中，需检查Logstash日志是否存在字段不匹配、ClickHouse连接失败等报错。

内容的提问来源于stack exchange，提问作者boycod3