Django集成Watchtower对接CloudWatch日志：无日志流创建

问题场景 #

我有一个使用Sentry进行错误追踪的Django项目，日志默认存储在服务器本地，希望通过Watchtower将日志转发到CloudWatch Logs。当前settings.py中的日志配置如下：

############# LOGGING ###########
from boto3 import client
import watchtower

logger_boto3_client = client(
    "logs",
    aws_access_key_id=AWS_ACCESS_KEY_ID,
    aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
    region_name=AWS_S3_REGION_NAME,
)

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "verbose": {
            "format": "[{levelname}] [{asctime}] [{pathname}] [{lineno}] [{message}]",
            "style": "{",
        },
    },
    "filters": {
        "debug_filter": {
            "()": "server_backend.log_settings.DebugFilter",
        },
        "info_filter": {
            "()": "server_backend.log_settings.InfoFilter",
        },
        "warning_filter": {
            "()": "server_backend.log_settings.WarningFilter",
        },
        "error_filter": {
            "()": "server_backend.log_settings.ErrorFilter",
        },
    },
    "handlers": {
        "debug_file": {
            "level": "DEBUG",
            "class": "logging.handlers.RotatingFileHandler",
            "filename": f"{BASE_DIR}/logs/debug.log",
            "formatter": "verbose",
            "maxBytes": 52428800,
            "backupCount": 10,
            "filters": ["debug_filter"],
        },
        "info_file": {
            "level": "INFO",
            "class": "logging.handlers.RotatingFileHandler",
            "filename": f"{BASE_DIR}/logs/info.log",
            "formatter": "verbose",
            "maxBytes": 52428800,
            "backupCount": 10,
            "filters": ["info_filter"],
        },
        "warning_file": {
            "level": "WARNING",
            "class": "logging.handlers.RotatingFileHandler",
            "filename": f"{BASE_DIR}/logs/warning.log",
            "formatter": "verbose",
            "maxBytes": 52428800,
            "backupCount": 10,
            "filters": ["warning_filter"],
        },
        "error_file": {
            "level": "ERROR",
            "class": "logging.handlers.RotatingFileHandler",
            "filename": f"{BASE_DIR}/logs/error.log",
            "maxBytes": 52428800,
            "backupCount": 10,
            "formatter": "verbose",
        },
        "watchtower": {
            "level": "DEBUG",
            "class": "watchtower.CloudWatchLogHandler",
            "boto3_client": logger_boto3_client,
            "log_group": "server_backend",
            "stream_name": "{strftime:%Y-%m-%d}",
            "formatter": "verbose",
            "create_log_group": True,  
            "create_log_stream": True, 
        },
    },
    "loggers": {             
        "server_backend": {
            "handlers": ["debug_file", "info_file", "warning_file", "error_file", "watchtower"],
            "level": "DEBUG",
            "propagate": True,
        },
        "django.request": {
            "handlers": ["debug_file", "info_file", "warning_file", "error_file"],
            "level": "DEBUG",
            "propagate": True,
        },
    },
}

异常现象 #

1. 在Django Shell中执行日志测试代码时，仅创建日志组但无日志流，且抛出警告：

import logging

logger = logging.getLogger("server_backend")
logger.info("This message should be logged without issues.")

输出：

.../env/lib/python3.10/site-packages/watchtower/__init__.py:437: WatchtowerWarning: Received message after logging system shutdown
  warnings.warn("Received message after logging system shutdown", WatchtowerWarning)
INFO:server_backend:This message should be logged without issues.

2. 手动在settings.py中添加代码直接创建handler时，能看到部分日志，但其他日志无法转发，触发自定义错误时仍会抛出相同警告。

问题原因分析 #

1. 提前初始化boto3客户端冲突：在Django加载settings.py阶段就创建boto3客户端，会导致客户端生命周期与日志系统生命周期不匹配。当Django进程（如Shell）退出时，日志系统先关闭，而提前创建的客户端已被销毁，后续待发送的日志无法正常提交，触发"Received message after logging system shutdown"警告。
2. 日志过滤器潜在拦截：自定义的DebugFilter、InfoFilter等若逻辑有误，可能意外拦截本该传递到Watchtower的日志；另外，需验证过滤器实现是否仅针对绑定的文件handler生效，未影响Watchtower的日志接收。
3. Django Shell的特殊生命周期：Shell会话退出时会快速关闭日志系统，Watchtower的后台异步日志发送线程可能还未完成日志提交，导致残留日志触发警告。

解决方案 #

1. 调整Watchtower Handler配置，移除提前初始化的boto3客户端 #

不要在settings.py开头提前创建boto3客户端，而是直接在Watchtower handler配置中传入AWS凭证参数，让Watchtower自行管理客户端生命周期：

修改后的LOGGING配置中，移除logger_boto3_client的创建代码，并更新watchtower handler：

############# LOGGING ###########
import watchtower

LOGGING = {
    # ... 其他配置保持不变 ...
    "handlers": {
        # ... 其他handler保持不变 ...
        "watchtower": {
            "level": "DEBUG",
            "class": "watchtower.CloudWatchLogHandler",
            "aws_access_key_id": AWS_ACCESS_KEY_ID,
            "aws_secret_access_key": AWS_SECRET_ACCESS_KEY,
            "region_name": AWS_S3_REGION_NAME,
            "log_group": "server_backend",
            "stream_name": "{strftime:%Y-%m-%d}",
            "formatter": "verbose",
            "create_log_group": True,  
            "create_log_stream": True,
            "flush_on_exit": True,  # 强制进程退出时flush所有待发送日志
        },
    },
    # ... 其他配置保持不变 ...
}

2. 验证自定义日志过滤器逻辑 #

确保DebugFilter、InfoFilter等过滤器仅过滤对应级别的日志，不会影响Watchtower handler的日志接收。例如，DebugFilter的实现应仅允许DEBUG级别日志通过（对应debug_file handler），而Watchtower handler未绑定过滤器，应能接收所有级别日志：

# 示例DebugFilter实现，确保逻辑正确
class DebugFilter(logging.Filter):
    def filter(self, record):
        return record.levelno == logging.DEBUG

3. 优化Django Shell测试方式 #

在Shell中测试时，可手动触发日志flush，避免退出时的警告：

import logging
from watchtower import CloudWatchLogHandler

logger = logging.getLogger("server_backend")
logger.info("Test log message")

# 手动flush所有Watchtower handler的日志
for handler in logger.handlers:
    if isinstance(handler, CloudWatchLogHandler):
        handler.flush()

4. 确认AWS权限与凭证正确性 #

再次确认IAM用户拥有logs:CreateLogGroup、logs:CreateLogStream、logs:PutLogEvents权限，且凭证未过期、区域配置正确。

验证效果 #

修改配置后，重启Django应用，正常触发日志（如接口请求、自定义错误），检查CloudWatch Logs中是否生成对应日志流及日志内容，同时确认无"Received message after logging system shutdown"警告。

内容的提问来源于stack exchange，提问作者hemant kumar chaudhary