Python日志：如何避免stderr与stdout中日志信息重复输出

我需要将INFO、WARNING级别日志输出到stdout，ERROR及以上级别输出到stderr，但当前配置下ERROR日志同时出现在stdout和stderr中，出现重复。以下是代码、配置及输出情况：

代码实现 #

import os
import subprocess
from typing import Union
import logging
import yaml
import logging.config

with open("logging.conf", "r") as f:
    config = yaml.safe_load(f.read())
    logging.config.dictConfig(config)


logger = logging.getLogger(__name__)    


def run(
    cmd: Union[str, list[str]],
    *,
    cwd: Union[str, os.PathLike[str], None] = None,
    shell: bool = False,
    check: bool = True,
    capture_output: bool = True,
    text: bool = True,
) -> subprocess.CompletedProcess[str]:
    logger.info("start run!!")
    return subprocess.run(cmd, check=True, capture_output=capture_output, text=text)


cmd = ["lxs"]


def main():
    try:
        output = run(cmd)
    except Exception as e:
        logger.error(e, exc_info=False)


main()

原日志配置（logging.conf） #

version: 1
formatters:
  simple:
    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
handlers:
  outconsole:
    class: logging.StreamHandler
    level: INFO
    formatter: simple
    stream: ext://sys.stdout
  errconsole:
    class: logging.StreamHandler
    level: WARNING
    formatter: simple
    stream: ext://sys.stderr

  file_handler:
    class: logging.FileHandler
    level: INFO
    formatter: simple
    filename: info.log
    encoding: utf8
    mode: w

loggers:
  sampleLogger:
    level: DEBUG
    handlers: [file_handler]
    propagate: no
root:
  level: INFO
  handlers: [errconsole, outconsole, file_handler]
  propagate: no

原执行结果 #

执行命令：python3 -m tests 1> output.log 2> err.log

• err.log内容：

2022-10-14 14:34:02,354 - __main__ - ERROR - [Errno 2] No such file or directory: 'lxs'

• output.log内容：

2022-10-14 14:34:02,353 - __main__ - INFO - start run!!
2022-10-14 14:34:02,354 - __main__ - ERROR - [Errno 2] No such file or directory: 'lxs'

• 控制台输出：

2022-10-14 14:48:33,178 - __main__ - INFO - start run!!
2022-10-14 14:48:33,179 - __main__ - ERROR - [Errno 2] No such file or directory: 'lxs'
2022-10-14 14:48:33,179 - __main__ - ERROR - [Errno 2] No such file or directory: 'lxs'

问题原因 #

当前配置中，outconsole handler的级别设为INFO，ERROR级别高于INFO，因此ERROR日志会被outconsole捕获输出；同时errconsole的级别是WARNING，ERROR也满足该级别条件，导致两个handler都输出ERROR日志，出现重复。

解决方法 #

通过添加日志过滤器，限制outconsole仅处理INFO和WARNING级别日志，拒绝ERROR及以上级别日志，同时调整errconsole的级别为ERROR，精准捕获目标日志。

1. 修改代码添加过滤器类 #

在导入logging模块后添加自定义过滤器：

class InfoWarningFilter(logging.Filter):
    def filter(self, record):
        # 只保留INFO和WARNING级别的日志
        return record.levelno in (logging.INFO, logging.WARNING)

2. 更新logging.conf配置 #

version: 1
formatters:
  simple:
    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
filters:
  info_warning_only:
    (): __main__.InfoWarningFilter  # 引用自定义过滤器类
handlers:
  outconsole:
    class: logging.StreamHandler
    level: INFO
    formatter: simple
    stream: ext://sys.stdout
    filters: [info_warning_only]  # 绑定过滤器
  errconsole:
    class: logging.StreamHandler
    level: ERROR  # 调整为ERROR，仅捕获ERROR及以上级别
    formatter: simple
    stream: ext://sys.stderr

  file_handler:
    class: logging.FileHandler
    level: INFO
    formatter: simple
    filename: info.log
    encoding: utf8
    mode: w

loggers:
  sampleLogger:
    level: DEBUG
    handlers: [file_handler]
    propagate: no
root:
  level: INFO
  handlers: [errconsole, outconsole, file_handler]
  propagate: no

验证修改后效果 #

重新执行命令python3 -m tests 1> output.log 2> err.log：

• output.log仅包含INFO和WARNING日志：

2022-10-14 14:34:02,353 - __main__ - INFO - start run!!

• err.log仅包含ERROR及以上日志：

2022-10-14 14:34:02,354 - __main__ - ERROR - [Errno 2] No such file or directory: 'lxs'

• 控制台输出不再出现重复的ERROR日志。

内容的提问来源于stack exchange，提问作者MattyOstrowsky