Filebeat多行模式解析含空行Python错误回溯异常问题

The Issue #

You're trying to collect full Python error traceback logs as single events in Logstash using Filebeat, but your current multiline configuration splits the log at the "Internal Server Error" line whenever there's an empty line above it. When there's no empty line, parsing works correctly—and even though your regex tested as expected in the Go playground, the Filebeat harvester still splits the logs incorrectly.

Full Log to Collect #

[pid: 17318|app: 0|req: 1/2] 10.14.206.28 (jaavedkhan) {60 vars in 1296 bytes} [Mon Dec 30 15:51:38 2019] GET /en/ => generated 27 bytes in 711 msecs (HTTP/1.1 500) 6 headers in 316 bytes (1 switches on core 0) 
Mon Dec 30 15:51:39 2019 - announcing my loyalty to the Emperor... 

Internal Server Error: /en/ 
Traceback (most recent call last): 
File "/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/core/handlers/exception.py", line 34, in inner 
response = get_response(request) 
File "/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/core/handlers/base.py", line 126, in _get_response 
response = self.process_exception_by_middleware(e, request) 
File "/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/core/handlers/base.py", line 124, in _get_response 
response = wrapped_callback(request, *callback_args, **callback_kwargs) 
File "/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/views/generic/base.py", line 68, in view 
return self.dispatch(request, *args, **kwargs) 
File "/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/views/generic/base.py", line 88, in dispatch 
return handler(request, *args, **kwargs) 
File "./core/views.py", line 31, in get 
1/0 
ZeroDivisionError: division by zero

Current Filebeat Configuration #

filebeat:
  inputs:
    - type: log
      paths:
        - "/var/log/uwsgi/vassals/dsr-incentives.log"
      fields_under_root: true
      multiline:
        pattern: '\[pid:\s*\d*\|app:'
        negate: true
        match: after
      fields:
        log_type: app-access
        appserver: uwsgi
        app: dsr-incentives
        server_name: server-name.domain.com

Example Published Event (Split Incorrectly) #

{
  "@timestamp": "2019-12-30T13:02:56.564Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.5.1"
  },
  "log": {
    "offset": 128736,
    "file": {
      "path": "/var/log/uwsgi/vassals/dsr-incentives.log"
    },
    "flags": [
      "multiline"
    ]
  },
  "appserver": "uwsgi",
  "server_name": "xyz",
  "log_type": "app-access",
  "host": {
    "name": "xyz"
  },
  "agent": {
    "hostname": "apps-1",
    "id": "d3417bc3-213c-4d5e-a9b5-2273178262d0",
    "version": "7.5.1",
    "name": "xyz",
    "type": "filebeat",
    "ephemeral_id": "125578d6-44d1-4103-94bc-a1d062091487"
  },
  "message": "Internal Server Error: /en/\nTraceback (most recent call last):\n File \"/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/core/handlers/exception.py\", line 34, in inner\n response = get_response(request)\n File \"/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/core/handlers/base.py\", line 126, in _get_response\n response = self.process_exception_by_middleware(e, request)\n File \"/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/core/handlers/base.py\", line 124, in _get_response\n response = wrapped_callback(request, *callback_args, **callback_kwargs)\n File \"/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/views/generic/base.py\", line 68, in view\n return self.dispatch(request, *args, **kwargs)\n File \"/opt/dsr-incentives/venv/lib/python3.5/site-packages/django/views/generic/base.py\", line 88, in dispatch\n return handler(request, *args, **kwargs)\n File \"./core/views.py\", line 31, in get\n 1/0\nZeroDivisionError: division by zero",
  "tags": [
    "filebeat"
  ],
  "input": {
    "type": "log"
  },
  "app": "dsr-incentives",
  "ecs": {
    "version": "1.1.0"
  }
}

Root Cause #

The empty line above "Internal Server Error" is triggering an unintended split. Your current pattern lacks a start anchor (^), meaning it could match the [pid:...|app: substring anywhere in a line (not just at the start). While this doesn't seem problematic at first, it can cause Filebeat's harvester to misinterpret empty lines as potential new entry boundaries, especially when combined with how multiline flushing works.

Solution #

Update your multiline configuration to include the start anchor, ensuring only lines that begin with the uwsgi [pid: prefix are treated as new log entries. Additionally, adding explicit timeout and max lines settings ensures the full traceback is grouped even if there are delays between lines:

Corrected Filebeat Multiline Configuration #

filebeat:
  inputs:
    - type: log
      paths:
        - "/var/log/uwsgi/vassals/dsr-incentives.log"
      fields_under_root: true
      multiline:
        pattern: '^\[pid:\s*\d*\|app:'  # Added ^ anchor to match only line starts
        negate: true
        match: after
        max_lines: 1000  # Ensure large enough to capture full tracebacks
        timeout: 5s      # Flush group if no new lines arrive after 5 seconds
      fields:
        log_type: app-access
        appserver: uwsgi
        app: dsr-incentives
        server_name: server-name.domain.com

How This Works #

• The ^ anchor ensures the pattern only matches lines starting with [pid:...|app:, correctly identifying new uwsgi request entries.
• negate: true tells Filebeat to append any line that doesn't match this pattern to the previous line.
• max_lines prevents truncating long tracebacks, and timeout ensures the full event is sent even if there's a pause in log output.

This configuration will group all lines related to a single request—including timestamp messages, empty lines, and full Python tracebacks—into one event for Logstash.

内容的提问来源于stack exchange，提问作者Javed