如何在Python中让structlog与logging均输出logfmt格式日志？

问题背景 #

希望根日志以logfmt格式输出，structlog也输出相同格式。尝试了structlog官方的JSON格式示例，但该方案不适用于logfmt。

首次尝试 #

代码 #

import logging
import sys

import logfmter
import structlog

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(logfmter.Logfmter())
root_logger = logging.getLogger()
root_logger.addHandler(handler)

structlog.get_logger("test").warning("hello")
logging.getLogger("test").warning("hello")

输出结果 #

2024-06-04 22:11:01 [warning  ] hello                         
at=WARNING msg=hello

可见structlog日志未采用logfmt格式。

添加logfmt渲染后的尝试 #

代码 #

import logging
import sys

import logfmter
import structlog

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(logfmter.Logfmter())
root_logger = logging.getLogger()
root_logger.addHandler(handler)

structlog.configure(
    processors=[
        structlog.stdlib.filter_by_level,
        structlog.stdlib.add_logger_name,
        structlog.stdlib.add_log_level,
        structlog.stdlib.PositionalArgumentsFormatter(),
        structlog.processors.TimeStamper(fmt='iso'),
        structlog.processors.StackInfoRenderer(),
        structlog.processors.format_exc_info,
        # ADD LOGFMT RENDER
        structlog.processors.LogfmtRenderer(),
    ],
    logger_factory=structlog.stdlib.LoggerFactory(),
    wrapper_class=structlog.stdlib.BoundLogger,
    cache_logger_on_first_use=True,
)

structlog.get_logger("test").warning("hello")
logging.getLogger("test").warning("hello")

输出结果 #

at=WARNING msg="event=hello logger=test level=warning timestamp=2024-06-04T13:12:54.637931Z"
at=WARNING msg=hello

预期格式 #

at=WARNING msg=hello logger=test level=warning timestamp=2024-06-04T13:12:54.637931Z

解决方案 #

要实现两者格式统一，核心是让structlog将处理后的日志数据传递给logging系统，由统一的logfmt formatter处理输出。调整后的代码如下：

最终代码 #

import logging
import sys

import logfmter
import structlog

# 配置logging的handler和logfmt formatter，指定输出字段与映射关系
handler = logging.StreamHandler(sys.stdout)
formatter = logfmter.Logfmter(
    keys=["at", "msg", "logger", "level", "timestamp"],
    mapping={
        "at": "levelname",
        "msg": "message",
        "logger": "name",
        "level": "levelname",
        "timestamp": "asctime"
    },
    datefmt="%Y-%m-%dT%H:%M:%S.%fZ"
)
handler.setFormatter(formatter)
root_logger = logging.getLogger()
root_logger.addHandler(handler)
root_logger.setLevel(logging.WARNING)  # 统一设置日志级别

# 配置structlog，将处理后的事件转交给logging系统
structlog.configure(
    processors=[
        structlog.stdlib.filter_by_level,
        structlog.stdlib.add_logger_name,
        structlog.stdlib.add_log_level,
        structlog.stdlib.PositionalArgumentsFormatter(),
        structlog.processors.TimeStamper(fmt="%Y-%m-%dT%H:%M:%S.%fZ", key="timestamp"),
        structlog.processors.StackInfoRenderer(),
        structlog.processors.format_exc_info,
        # 将structlog事件转换为logging可识别的关键字参数
        structlog.stdlib.render_to_log_kwargs,
    ],
    logger_factory=structlog.stdlib.LoggerFactory(),
    wrapper_class=structlog.stdlib.BoundLogger,
    cache_logger_on_first_use=True,
)

# 测试输出
structlog.get_logger("test").warning("hello")
logging.getLogger("test").warning("hello")

输出结果 #

at=WARNING msg=hello logger=test level=WARNING timestamp=2024-06-04T13:12:54.637931Z
at=WARNING msg=hello logger=test level=WARNING timestamp=2024-06-04T13:12:54.637931Z

关键调整说明 #

• 移除structlog独立渲染：去掉structlog.processors.LogfmtRenderer()，改用structlog.stdlib.render_to_log_kwargs，让structlog把处理后的字段（如logger、timestamp）传递给logging。
• 统一logfmt formatter配置：通过keys定义输出字段，mapping关联logging内置字段与logfmt键名，同时指定统一的时间格式，确保两者输出一致。
• 设置日志级别：给root_logger指定级别，避免默认级别导致日志不输出的问题。

内容的提问来源于stack exchange，提问作者rzlvmp