Gunicorn日志重复问题及自定义Stackdriver JSON日志模块咨询

First, let's break this into two clear parts: resolving Gunicorn's duplicate logging issue, and refining your custom formatter to properly fit Stackdriver's JSON requirements.

Step 1: Why Logs Duplicate #

Gunicorn sets up its own default log handlers, and most web frameworks (like Flask/Django) do the same. Duplicates happen when:

• Your app's logs propagate up to Gunicorn's root logger, triggering both handlers to process the same message
• Accidentally adding multiple handlers to the same logger instance

Step 2: Complete Your Stackdriver JSON Formatter #

Your existing code is a solid start—let's finish it to match Stackdriver's required fields (like severity and timestamp). Save this as stackdriver_logging.py:

import logging
import sys
from pythonjsonlogger import jsonlogger
from datetime import datetime
import iso8601

class _MaxLevelFilter(object):
    def __init__(self, highest_log_level):
        self._highest_log_level = highest_log_level
    def filter(self, log_record):
        return log_record.levelno <= self._highest_log_level

class StackdriverJsonFormatter(jsonlogger.JsonFormatter):
    def __init__(self, **kwargs):
        # Base fields to include in every log
        super().__init__(
            fmt="%(message)s %(levelname)s %(asctime)s %(module)s %(lineno)d",
            **kwargs
        )

    def formatRecord(self, record):
        # Override to shape logs for Stackdriver compatibility
        log_dict = super().formatRecord(record)
        
        # Map to Stackdriver's required and recommended fields
        stackdriver_log = {
            "message": log_dict["message"],
            "severity": log_dict["levelname"],
            "timestamp": iso8601.to_string(datetime.fromtimestamp(record.created)),
            "module": log_dict["module"],
            "line_number": log_dict["lineno"]
        }
        
        # Add exception details if the log includes an error trace
        if record.exc_info:
            stackdriver_log["exception"] = self.formatException(record.exc_info)
        
        # Include any custom extra fields attached to the log record
        if hasattr(record, "extra"):
            stackdriver_log.update(record.extra)
        
        return stackdriver_log

Step 3: Configure Gunicorn to Stop Duplicates & Use the Formatter #

Create a Gunicorn config file (e.g., gunicorn.conf.py) to take full control of logging:

import logging
import sys
from stackdriver_logging import StackdriverJsonFormatter, _MaxLevelFilter

def setup_logging():
    # Grab Gunicorn's root logger instance
    gunicorn_logger = logging.getLogger("gunicorn")
    
    # Remove all default handlers to eliminate duplicate processing
    for handler in gunicorn_logger.handlers[:]:
        gunicorn_logger.removeHandler(handler)
    
    # Split logs to follow standard practice: INFO+ below to stdout, WARNING+ to stderr
    stdout_handler = logging.StreamHandler(sys.stdout)
    stdout_handler.addFilter(_MaxLevelFilter(logging.INFO))
    
    stderr_handler = logging.StreamHandler(sys.stderr)
    stderr_handler.setLevel(logging.WARNING)
    
    # Attach our Stackdriver formatter to both handlers
    formatter = StackdriverJsonFormatter()
    stdout_handler.setFormatter(formatter)
    stderr_handler.setFormatter(formatter)
    
    # Add the configured handlers back to Gunicorn's logger
    gunicorn_logger.addHandler(stdout_handler)
    gunicorn_logger.addHandler(stderr_handler)
    
    # Disable log propagation to prevent logs from bubbling up to the root logger
    gunicorn_logger.propagate = False

# Run the logging setup when Gunicorn starts
setup_logging()

# Your standard Gunicorn server configs
bind = "0.0.0.0:8000"
workers = 2
timeout = 30

Step 4: Configure Your App to Avoid Duplicates #

Make sure your web app doesn't add extra handlers or propagate logs to Gunicorn.

For Flask Apps: #

from flask import Flask
import logging
from stackdriver_logging import StackdriverJsonFormatter

app = Flask(__name__)

# Stop app logs from propagating to Gunicorn's logger
app.logger.propagate = False

# Remove Flask's default handlers to avoid conflicts
for handler in app.logger.handlers[:]:
    app.logger.removeHandler(handler)

# Attach our Stackdriver formatter to the app's logger
handler = logging.StreamHandler()
handler.setFormatter(StackdriverJsonFormatter())
app.logger.addHandler(handler)
app.logger.setLevel(logging.INFO)

For Django Apps: #

Update your settings.py logging config:

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "stackdriver": {
            "()": "stackdriver_logging.StackdriverJsonFormatter",
        },
    },
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "formatter": "stackdriver",
        },
    },
    "root": {
        "handlers": ["console"],
        "level": "INFO",
        "propagate": False,
    },
}

Step 5: Verify the Setup #

Start Gunicorn with your config:

gunicorn --config gunicorn.conf.py your_app:app

Check your logs—you should see single, valid JSON entries for each log, with fields Stackdriver recognizes (severity, timestamp, etc.), no duplicates.

Key Dependencies #

Install the required packages first:

pip install python-json-logger iso8601

内容的提问来源于stack exchange，提问作者Mauricio