使用Fluent-Bit向Syslog发送Containerd容器日志时出现"failed to flush chunk"错误

把容器运行时从Docker切换到Containerd后，尝试用Fluent-Bit将容器日志发送至Syslog，多次调整配置都没解决，始终出现failed to flush chunk错误。

配置文件 #

[SERVICE]
    Flush 1
    Daemon Off
    Log_Level debug
    Parsers_File /etc/fluent-bit/parsers.conf

[INPUT]
    Name tail
    Path /var/log/containers/*.log
    Parser cri
    Tag kube.*
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On

[OUTPUT]
    name                 syslog
    host                 127.0.0.1
    match                *
    port                 514
    mode                 udp
    syslog_format        rfc5424
    syslog_maxsize       2048
    syslog_message_key   message
    syslog_severity_key  severity
    syslog_facility_key  facility
    syslog_hostname_key  hostname
    syslog_appname_key   appname
    syslog_procid_key    procid
    syslog_msgid_key     msgid
    syslog_sd_key        sd

[FILTER]
    Name kubernetes
    Match kube.*
    Merge_Log On
    Merge_Log_Trim On
    Labels Off
    Annotations Off
    K8S-Logging.Parser Off
    K8S-Logging.Exclude Off

调试模式下的Fluent-Bit日志片段 #

Nov 03 22:27:24 apicdev1224 fluent-bit[3247533]: [2022/11/03 22:27:24] [debug] [input:tail:tail.0] inode=1290451 events: IN_MODIFY
Nov 03 22:27:24 apicdev1224 fluent-bit[3247533]: [2022/11/03 22:27:24] [debug] [filter:kubernetes:kubernetes.0] Send out request to API Server for pods information
Nov 03 22:27:24 apicdev1224 fluent-bit[3247533]: [2022/11/03 22:27:24] [debug] [input chunk] update output instances with new chunk size diff=202
Nov 03 22:27:24 apicdev1224 fluent-bit[3247533]: [2022/11/03 22:27:24] [debug] [task] created task=0x7f0efd03dbd0 id=6 OK
Nov 03 22:27:24 apicdev1224 fluent-bit[3247533]: [2022/11/03 22:27:24] [debug] [task] created task=0x7f0efd03dcb0 id=8 OK
Nov 03 22:27:24 apicdev1224 fluent-bit[3247533]: [2022/11/03 22:27:24] [debug] [input:tail:tail.0] inode=1290451 events: IN_MODIFY
Nov 03 22:27:24 apicdev1224 fluent-bit[3247533]: [2022/11/03 22:27:24] [debug] [filter:kubernetes:kubernetes.0] Send out request to API Server for pods information
Nov 03 22:27:24 apicdev1224 fluent-bit[3247533]: [2022/11/03 22:27:24] [debug] [input chunk] update output instances with new chunk size diff=184
Nov 03 22:27:24 apicdev1224 fluent-bit[3247533]: [2022/11/03 22:27:24] [debug] [out flush] cb_destroy coro_id=8
Nov 03 22:27:24 apicdev1224 fluent-bit[3247533]: [2022/11/03 22:27:24] [debug] [out flush] cb_destroy coro_id=9
Nov 03 22:27:24 apicdev1224 fluent-bit[3247533]: [2022/11/03 22:27:24] [debug] [retry] new retry created for task_id=6 attempts=1
Nov 03 22:27:24 apicdev1224 fluent-bit[3247533]: [2022/11/03 22:27:24] [ warn] [engine] failed to flush chunk '3247533-1667514444.161129247.flb', retry in 9 seconds: task_id=6, input=tail.0 > output=syslog.0 (out_id=0)
Nov 03 22:27:24 apicdev1224 fluent-bit[3247533]: [2022/11/03 22:27:24] [debug] [retry] new retry created for task_id=8 attempts=1

版本信息 #

• fluent-bit version: 2.0.3
• containerd version: 1.5.11-1

解决步骤 #

1. 检查Syslog服务状态与端口可达性 #

• 用nc -u -zv 127.0.0.1 514测试UDP端口连通性，确认Syslog服务在监听该端口
• 查看Syslog服务状态：systemctl status rsyslog（或对应syslog服务名称），确保服务处于运行状态

2. 修正Syslog输出配置的字段缺失问题 #

Containerd解析后的日志大概率没有你配置中指定的severity、facility等字段，导致Fluent-Bit无法构造合法的RFC5424日志包，进而发送失败。调整配置如下：

[OUTPUT]
    name                 syslog
    host                 127.0.0.1
    match                *
    port                 514
    mode                 udp
    syslog_format        rfc5424
    syslog_maxsize       2048
    syslog_message_key   message
    # 直接设置默认值，替代依赖日志中不存在的字段
    syslog_severity      6
    syslog_facility      1
    syslog_hostname      ${HOSTNAME}
    syslog_appname       fluent-bit

3. 验证Containerd日志解析结果 #

临时添加stdout输出，查看解析后的日志字段结构，确认message等关键字段存在：

[OUTPUT]
    name stdout
    match *
    format json_lines

启动Fluent-Bit后，检查终端输出的JSON内容，确保日志结构符合预期。

4. 调整Fluent-Bit刷新与重试参数 #

过小的Flush间隔可能导致频繁发送请求，触发重试机制。修改[SERVICE]段配置：

[SERVICE]
    Flush 5
    Daemon Off
    Log_Level debug
    Parsers_File /etc/fluent-bit/parsers.conf
    Retry_Limit False
    Retry_Wait 3

5. 检查SELinux与防火墙限制 #

• 临时关闭SELinux测试：setenforce 0，若问题解决，需添加SELinux规则允许Fluent-Bit访问Syslog端口
• 检查防火墙规则：iptables -L -n | grep 514，确保UDP 514端口允许本地访问

内容的提问来源于stack exchange，提问作者Kenny Nguyen