Fluent-bit传输AWS EKS Java应用日志时截断丢弃问题求助

我正在使用Fluent-bit将AWS EKS中的日志发送至AWS CloudWatch，多数日志能正常传输，但大日志会出现异常：起初被截断，最终直接被丢弃。

相关错误日志 #

[2023/12/05 14:10:54] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=77603133 watch_fd=5
[2023/12/05 14:30:14] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] [size=294805] Truncating event which is larger than max size allowed by CloudWatch
[2023/12/05 14:30:19] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] [size=364675] Truncating event which is larger than max size allowed by CloudWatch
[2023/12/05 14:30:24] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] [size=523322] Truncating event which is larger than max size allowed by CloudWatch
[2023/12/05 14:30:29] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Discarding massive log record
[2023/12/05 14:30:29] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] [size=836367] Truncating event which is larger than max size allowed by CloudWatch
[2023/12/05 14:30:34] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] [size=554279] Truncating event which is larger than max size allowed by CloudWatch
[2023/12/05 14:30:39] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] [size=731990] Truncating event which is larger than max size allowed by CloudWatch
[2023/12/05 14:30:39] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] [size=793332] Truncating event which is larger than max size allowed by CloudWatch

当前配置 #

fluent-bit.conf #

[SERVICE]
    Flush                     5
    Grace                     30
    Log_Level                 info
    Daemon                    off
    Parsers_File              parsers.conf
    HTTP_Server               ${HTTP_SERVER}
    HTTP_Listen               0.0.0.0
    HTTP_Port                 ${HTTP_PORT}
    storage.path              /var/fluent-bit/state/flb-storage/
    storage.sync              normal
    storage.checksum          off
    storage.max_chunks_up     128
    storage.backlog.mem_limit 5M
    scheduler.cap             30

@INCLUDE application-log.conf
@INCLUDE dataplane-log.conf
@INCLUDE host-log.conf

application-log.conf #

[INPUT]
    Name                tail
    Tag                 application.*
    Exclude_Path        /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*
    Path                /var/log/containers/*.log
    multiline.parser    java, docker, cri
    DB                  /var/fluent-bit/state/flb_container.db
    Mem_Buf_Limit       50MB
    Skip_Long_Lines     Off
    Refresh_Interval    5
    Rotate_Wait         30
    storage.type        filesystem
    Read_from_Head      ${READ_FROM_HEAD}

[INPUT]
    Name                tail
    Tag                 application.*
    Path                /var/log/containers/cloudwatch-agent*
    multiline.parser    docker, cri
    DB                  /var/fluent-bit/state/flb_cwagent.db
    Mem_Buf_Limit       5MB
    Skip_Long_Lines     Off
    Refresh_Interval    30
    Read_from_Head      ${READ_FROM_HEAD}

[FILTER]
    Name                kubernetes
    Match               application.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_Tag_Prefix     application.var.log.containers.
    Merge_Log           On
    Merge_Log_Key       log_processed
    K8S-Logging.Parser  On
    K8S-Logging.Exclude Off
    Labels              On
    Annotations         Off

[OUTPUT]
    Name                cloudwatch_logs
    Match               application.*
    region              ${AWS_REGION}
    log_group_name      /aws/containerinsights/${CLUSTER_NAME}/application
    log_stream_name     $(kubernetes['container_name'])-$(kubernetes['pod_name'])
    log_stream_template $kubernetes['container_name'].$kubernetes['pod_name']
    auto_create_group   On
    Retry_Limit         False

parsers.conf #

[MULTILINE_PARSER]
    name                multiline-regex
    type                regex
    flush_timeout       5
    # rules |   state name  | regex pattern                  | next state
    # ------|---------------|--------------------------------------------
    rule      "start_state"   "/(Dec \d+ \d+\:\d+\:\d+)(.*)/"  "cont"
    rule      "cont"          "/^\s+at.*/"                     "cont"

[PARSER]
    Name                docker
    Format              json
    Time_Key            time
    Time_Format         %Y-%m-%dT%H:%M:%S.%LZ

[PARSER]
    Name                syslog
    Format              regex
    Regex               ^(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
    Time_Key            time
    Time_Format         %b %d %H:%M:%S

[PARSER]
    Name                container_firstline
    Format              regex
    Regex               (?<log>(?<="log":"\S(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=})
    Time_Key            time
    Time_Format         %Y-%m-%dT%H:%M:%S.%LZ

[PARSER]
    Name                cwagent_firstline
    Format              regex
    Regex               (?<log>(?<="log":"\d{4}[\/-]\d{1,2}[\/-]\d{1,2}[ T]\d{2}:\d{2}:\d{2}(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=})
    Time_Key            time
    Time_Format         %Y-%m-%dT%H:%M:%S.%LZ

已尝试的操作 #

• 分别使用cloudwatch和cloudwatch_logs插件发送日志，均出现相同的截断/丢弃问题
• 尝试过内置Java解析器及多种自定义多行解析器，问题未解决
• 观察到Pod运行一段时间后才出现错误，推测可能与缓冲内存配置有关

内容的提问来源于stack exchange，提问作者MathiasCodes