requests调用时urllib3.connectionpool日志脱敏过滤器失效问题排查

我想给requests依赖的urllib3日志器添加Filter，脱敏日志中的敏感信息，但调用requests.get()时，urllib3.connectionpool日志器上的过滤器完全没生效。

可复现示例 #

import logging
import re
import requests


class Redactor(logging.Filter):
    """Filter subclass to redact patterns from logs."""
    redact_replacement_string = "<REDACTED_INFO>"

    def __init__(self, patterns: list[re.Pattern] = None):
        super().__init__()
        self.patterns = patterns or list()

    def filter(self, record: logging.LogRecord) -> bool:
        """
        Overriding the original filter method to redact, rather than filter.
        :return: Always true - i.e. always apply filter
        """
        for pattern in self.patterns:
            record.msg = pattern.sub(self.redact_replacement_string, record.msg)
        return True

# 设置日志级别
urllib_logger = logging.getLogger("urllib3.connectionpool")
urllib_logger.setLevel("DEBUG")

# 添加处理器
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("logger name: {name} | message: {message}", style="{"))
urllib_logger.addHandler(handler)

# 添加过滤器
urllib_logger.info("Sensitive string before applying filter: www.google.com")
sensitive_patterns = [re.compile(r"google")]
redact_filter = Redactor(sensitive_patterns)
urllib_logger.addFilter(redact_filter)
urllib_logger.info("Sensitive string after applying filter: www.google.com")

# 发起应使用已过滤日志器的请求
requests.get("https://www.google.com")

# 检查日志器是否被重新配置
urllib_logger.info("Sensitive string after request: www.google.com")

代码执行结果 #

Handler对所有日志字符串生效，但Filter未处理requests.get()输出的日志：

logger name: urllib3.connectionpool | message: Sensitive string before applying filter: www.google.com
logger name: urllib3.connectionpool | message: Sensitive string after applying filter: www.<REDACTED_INFO>.com
logger name: urllib3.connectionpool | message: Starting new HTTPS connection (1): www.google.com:443
logger name: urllib3.connectionpool | message: https://www.google.com:443 "GET / HTTP/1.1" 200 None
logger name: urllib3.connectionpool | message: Sensitive string after request: www.<REDACTED_INFO>.com

预期效果 #

所有包含敏感模式“google”的日志都被脱敏：

logger name: urllib3.connectionpool | message: Sensitive string before applying filter: www.google.com
logger name: urllib3.connectionpool | message: Sensitive string after applying filter: www.<REDACTED_INFO>.com
logger name: urllib3.connectionpool | message: Starting new HTTPS connection (1): www.<REDACTED_INFO>.com:443
logger name: urllib3.connectionpool | message: https://www.<REDACTED_INFO>.com:443 "GET / HTTP/1.1" 200 None
logger name: urllib3.connectionpool | message: Sensitive string after request: www.<REDACTED_INFO>.com

已尝试方案 #

• 将同一Filter应用于root日志器、urllib3日志器及所有现有日志器，结果相同：

all_loggers = [logger for logger in logging.root.manager.loggerDict.values()
               if not isinstance(logger, logging.PlaceHolder)]

for logger in all_loggers:
    logger.addFilter(redact_filter)

• 将Filter应用于Handler而非Logger，问题依旧。
• 已知可通过继承Formatter类实现脱敏，但认为格式化与脱敏是独立功能，希望分离实现，同时想理解日志模块的相关逻辑。

问题原因与解决方案 #

原因 #

urllib3的日志记录采用带占位符的模板消息+参数列表的方式，比如日志消息是"Starting new HTTPS connection (1): %s:%s"，实际的域名和端口是通过record.args传递的。你的过滤器只修改了record.msg（模板），但没有处理record.args中的实际敏感数据，所以最终格式化后的日志还是会显示原始敏感信息。

修正后的Filter实现 #

修改Redactor类，同时处理record.msg和record.args中的敏感内容：

import logging
import re
import requests


class Redactor(logging.Filter):
    """Filter subclass to redact patterns from logs."""
    redact_replacement_string = "<REDACTED_INFO>"

    def __init__(self, patterns: list[re.Pattern] = None):
        super().__init__()
        self.patterns = patterns or list()

    def _redact_string(self, s: str) -> str:
        """Helper to redact a single string."""
        for pattern in self.patterns:
            s = pattern.sub(self.redact_replacement_string, s)
        return s

    def filter(self, record: logging.LogRecord) -> bool:
        # 处理消息模板
        record.msg = self._redact_string(record.msg)
        
        # 处理参数列表中的字符串元素
        if record.args:
            new_args = []
            for arg in record.args:
                if isinstance(arg, str):
                    new_args.append(self._redact_string(arg))
                else:
                    new_args.append(arg)
            record.args = tuple(new_args)
        return True

# 后续配置代码不变
urllib_logger = logging.getLogger("urllib3.connectionpool")
urllib_logger.setLevel("DEBUG")

handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("logger name: {name} | message: {message}", style="{"))
urllib_logger.addHandler(handler)

sensitive_patterns = [re.compile(r"google")]
redact_filter = Redactor(sensitive_patterns)
urllib_logger.addFilter(redact_filter)

urllib_logger.info("Sensitive string before applying filter: www.google.com")
urllib_logger.info("Sensitive string after applying filter: www.google.com")

requests.get("https://www.google.com")

urllib_logger.info("Sensitive string after request: www.google.com")

执行结果 #

修正后，所有日志都会正确脱敏：

logger name: urllib3.connectionpool | message: Sensitive string before applying filter: www.google.com
logger name: urllib3.connectionpool | message: Sensitive string after applying filter: www.<REDACTED_INFO>.com
logger name: urllib3.connectionpool | message: Starting new HTTPS connection (1): www.<REDACTED_INFO>.com:443
logger name: urllib3.connectionpool | message: https://www.<REDACTED_INFO>.com:443 "GET / HTTP/1.1" 200 None
logger name: urllib3.connectionpool | message: Sensitive string after request: www.<REDACTED_INFO>.com

额外说明 #

• 日志模块的执行流程：日志记录时先创建LogRecord，然后依次经过日志器的过滤器、处理器的过滤器，最后由处理器的格式化器将msg和args组合成最终日志字符串。
• 如果你需要处理更复杂的参数类型（比如非字符串的对象），可以在_redact_string中添加类型判断和转换逻辑。

内容的提问来源于stack exchange，提问作者Mart Roben