Filebeat能否发送日志至Fluentd？配置报错求助

一、Docker环境下的连接拒绝错误 #

Filebeat持续报错：

2023-11-08 17:02:31 2023-11-08T10:02:31.482Z    ERROR   [publisher_pipeline_output]     pipeline/output.go:154  Failed to connect to backoff(async(tcp://127.0.0.1:5044)): dial tcp 127.0.0.1:5044: connect: connection refused
2023-11-08 17:02:31 2023-11-08T10:02:31.482Z    INFO    [publisher_pipeline_output]     pipeline/output.go:145  Attempting to reconnect to backoff(async(tcp://127.0.0.1:5044)) with 11 reconnect attempt(s)

相关配置文件 #

filebeat.yml

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  paths:
    - /log/test/*.log
    
output.logstash:
  # The Logstash hosts
  hosts: ["127.0.0.1:5044"]

fluent.conf

<source>
  @type beats
  metadata_as_tag  
  # port 5044
  # bind localhost
</source>


<match *beat>
  @type copy
  <store>
    @type file
    path /output/test
    add_path_suffix true
    path_suffix ".txt"
  </store>
  <store>
    @type stdout
  </store>
</match> 

docker-compose.yml

version: "3"
services:
  fluentd:
    container_name: fluentd
    build: 
      context: ./fluentd
      dockerfile: Dockerfile
    volumes:
      - ./fluentd/conf:/fluentd/etc
      - ./fluentd/buffer:/fluentd/buffer
      - ./fluentd/data:/output:rw
    ports:
      - "24224:24224"
      - "24224:24224/udp"
      - "1514:1514/udp"
      - "13543:13543/udp"
      - "5044:5044"
    networks:
      - fluent
  filebeat:
    container_name: filebeat
    build:
      context: ./filebeat
      dockerfile: Dockerfile
    volumes: 
      - ./filebeat/log:/log/test/:rw
    networks:
      - fluent
networks:
  fluent:
    driver: bridge

二、本地直接运行Filebeat的错误 #

2023-11-09T08:15:24+00:00   fluent.error    {"error":"undefined method `[]' for nil:NilClass","message":"unexpected error error=\"undefined method `[]' for nil:NilClass\""}

解决方案 #

解决Docker环境下的连接拒绝问题 #

1. 修改Filebeat的目标地址：Docker容器内的127.0.0.1指向容器自身，而非宿主机或其他容器。在同一fluent网络下，直接使用Fluentd的服务名fluentd作为地址，修改filebeat.yml：

   output.logstash:
     hosts: ["fluentd:5044"]
   

2. 修正Fluentd的Beats源配置：取消注释端口和绑定地址，将bind设为0.0.0.0（监听容器所有网络接口，允许外部容器访问），修改fluent.conf的<source>块：

   <source>
     @type beats
     metadata_as_tag  
     port 5044
     bind 0.0.0.0
   </source>
   

解决本地运行的undefined method [] for nil:NilClass错误 #

这个错误源于fluent-plugin-beats处理元数据时的空值问题，可通过以下方式解决：

1. 禁用metadata_as_tag：如果不需要将元数据转为标签，直接在Fluentd配置中移除该配置项；
2. 升级插件版本：旧版fluent-plugin-beats存在该bug，执行gem update fluent-plugin-beats升级至最新版本；
3. 确保Filebeat发送元数据：在Filebeat配置中添加元数据字段，比如：

   filebeat.inputs:
   - type: filestream
     id: my-filestream-id
     paths:
       - /log/test/*.log
     fields:
       source_tag: "my-log"
     fields_under_root: true
   

内容的提问来源于stack exchange，提问作者Tanin Imanothai