Python Logging模块extra属性与自定义格式化器兼容问题排查

问题描述 #

在Python项目中测试自定义logger时，extra参数始终无法生效：既无法在格式字符串中配置该属性，自定义格式化器也无法读取它。手动传入extra或依赖LoggerAdapter自动添加的场景均失效，在配置文件中添加hostname/ip字段时会抛出ValueError: Formatting field not found in record: 'hostname'错误，其余日志功能正常。

错误点分析 #

1. LoggerAdapter未正确赋值
   创建LoggerAdapter后没有将其赋值给原logger变量，后续日志调用仍然使用原始的logging.Logger实例，而非带额外参数的Adapter。
2. 自定义Formatter对extra的处理逻辑错误
   LoggerAdapter会将extra中的键值对直接合并到LogRecord的__dict__属性中，而非存入record.extra字段。原代码中通过getattr(record, "extra", {})无法获取到传入的额外参数。
3. 格式字符串未处理缺失字段
   配置文件中直接使用%(hostname)s这类字段时，若存在未携带该参数的日志记录，会触发字段缺失错误。

修正步骤 #

1. 正确初始化LoggerAdapter #

修改main.py中的logger初始化代码，将创建的LoggerAdapter赋值给logger变量：

import logging
import socket
import subprocess

# 初始化logger并包装为LoggerAdapter
logger = logging.getLogger(__name__)
hostname = socket.gethostname()
host_ip = subprocess.run(
    [r"ip -4 addr show ztjlhzlhyj | grep -oP '(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})'"],
    capture_output=True, text=True, shell=True
).stdout.splitlines()[0].strip()
# 将LoggerAdapter赋值给logger，后续调用使用该实例
logger = logging.LoggerAdapter(logger, extra={"hostname": hostname, "ip": host_ip})

# 手动传入extra的示例（此时可省略，Adapter会自动添加，若需覆盖可保留）
logger.debug("Loading env...")

# 依赖LoggerAdapter自动添加extra的示例
try:
    raise Exception("Test error")
except Exception as e:
    logger.error(e)

2. 修改自定义Formatter的extra处理逻辑 #

调整maxlogger.py中MaxJSONFormatter的_prepare_log_dict方法，直接读取LogRecord.__dict__中的非内置属性：

import json
from datetime import datetime, timezone
import logging
from typing import Optional

LOG_RECORD_BUILTIN_ATTRS = {
    "args", "asctime", "created", "exc_info", "exc_text", "filename",
    "funcName", "levelname", "levelno", "lineno", "module", "msecs",
    "message", "msg", "name", "pathname", "process", "processName",
    "relativeCreated", "stack_info", "thread", "threadName", "taskName",
}

class MaxJSONFormatter(logging.Formatter):
    def __init__(self, *, fmt_keys: Optional[dict[str, str]] = None):
        super().__init__()
        self.fmt_keys = fmt_keys if fmt_keys is not None else dict()

    def format(self, record: logging.LogRecord) -> str:
        message = self._prepare_log_dict(record)
        return json.dumps(message, default=str)

    def _prepare_log_dict(self, record: logging.LogRecord) -> dict:
        always_fields = {
            "message": record.getMessage(),
            "timestamp": datetime.fromtimestamp(
                record.created, tz=timezone.utc
            ).isoformat()
        }

        if record.exc_info is not None:
            always_fields["exc_info"] = self.formatException(record.exc_info)

        if record.stack_info is not None:
            always_fields["stack_info"] = self.formatStack(record.stack_info)

        # 构建基础字段
        message = {
            key: msg_val
            if (msg_val := always_fields.pop(val, None)) is not None
            else getattr(record, val)
            for key, val in self.fmt_keys.items()
        }

        # 添加始终字段
        message.update(always_fields)

        # 添加所有非内置的额外属性（包括extra传入的字段）
        for key, val in record.__dict__.items():
            if key not in LOG_RECORD_BUILTIN_ATTRS and key not in message:
                message[key] = val

        return message

3. 配置文件中处理缺失字段 #

在格式字符串中为hostname/ip添加默认值，避免因日志未携带该字段报错：

{
  "version": 1,
  "disable_existing_loggers": false,
  "formatters": {
    "simple": {
      "format": "[%(levelname)s]: %(message)s"
    },
    "detailed": {
      "format": "[%(levelname)s|%(module)s|%(lineno)d|%(hostname)s:-|%(ip)s:-] - %(asctime)s: %(message)s",
      "datefmt": "%Y-%m-%dT%H:%M:%S%z"
    },
    "json": {
      "()": "maxlogger.MaxJSONFormatter",
      "fmt_keys": {
        "level": "levelname",
        "message": "message",
        "timestamp": "timestamp",
        "logger": "name",
        "module": "module",
        "function": "funcName",
        "line": "lineno",
        "thread_name": "threadName"
      }
    }
  },
  "handlers": {
    "stderr": {
      "class": "logging.StreamHandler",
      "level": "WARNING",
      "formatter": "detailed",
      "stream": "ext://sys.stderr"
    },
    "file": {
      "class": "logging.handlers.RotatingFileHandler",
      "level": "DEBUG",
      "formatter": "detailed",
      "filename": "logs/debug.log",
      "maxBytes": 104857600,
      "backupCount": 3
    },
    "db": {
      "()": "maxlogger.MaxDBHandler",
      "level": "WARNING",
      "formatter": "json"
    },
    "queue": {
      "()": "maxlogger.MaxQueueHandler",
      "handlers": [
        "stderr",
        "file",
        "db"
      ],
      "respect_handler_level": true
    }
  },
  "loggers": {
    "root": {
      "level": "DEBUG",
      "handlers": [
        "queue"
      ]
    }
  }
}

关键说明 #

• LoggerAdapter的作用是为所有通过它发起的日志调用自动添加extra参数，无需每次手动传入。
• LogRecord的额外属性会直接存入__dict__，无需从record.extra读取。
• 格式字符串中%(field)s:-的语法表示当field不存在时，使用-作为默认值，避免报错。

内容的提问来源于stack exchange，提问作者LaZoR_Bear