使用Python的newrelic_telemetry_sdk无法在New Relic添加日志自定义字段

问题描述 #

使用New Relic时，无法通过任何Handler发送包含%(asctime)s-%(correlation_id)s-%(levelname)s-%(message)s-格式的自定义字段日志。使用newrelic_telemetry_sdk的LogClient能在控制台查看普通日志，但尝试StreamHandler、SysLogHandler、FileHandler等多种Handler后，correlation_id等自定义参数始终无法出现在New Relic日志面板中。

现有代码片段 #

初始日志配置代码 #

import logging
from newrelic_telemetry_sdk import Log, LogClient
from fastapi import FastAPI, Response
from logging import handlers

app = FastAPI()
logger = logging.getLogger(__name__)
_log_format = "%(asctime)s-%(correlation_id)s-%(levelname)s-%(message)s-%(http_method)s:%(http_endpoint)s-%(funcName)s-%(lineno)d]-%(user_id)s-%(client_ip)s-%(user_agent)s"

log_client = LogClient('#Added my new relic liscence key in here')

# 尝试过的各类Handler，均未生效
handler = logging.StreamHandler()
logger.addHandler(handler)

# handler = logging.handlers.SysLogHandler(address=('log-api.newrelic.com', 443))
# logger.addHandler(handler)

# formatter = NewRelicContextFormatter()
# fh = logging.FileHandler("/root/.newrelic/newrelic-cli.log", mode="a")
# logger.addHandler(fh)

# file = handlers.RotatingFileHandler("/root/.newrelic/newrelic-cli.log", mode="a")
# file.setFormatter(NewRelicContextFormatter())
# logger.addHandler(file)

# handler = logging.NullHandler()
# logger.addHandler(handler)

后续添加的Filter（Update_2） #

import newrelic
import newrelic.agent
import logging

class HeaderFilter(logging.Filter):
    def __init__(self):
        super().__init__()

    def filter(self, record):
        setattr(record, "correlation_id", getattr(self, "correlation_id", None))
        setattr(record, "user_id", getattr(self, "user_id", None))
        setattr(record, "user_agent", getattr(self, "user_agent", None))

        list1 = []
        for key, value in record.__dict__.items():
            list1.append((key, value))
            newrelic.agent.add_custom_attributes(list1)
        return True
        
logger.addFilter(HeaderFilter())

后续添加的中间件（Update_3） #

def global_middleware(request, call_next):

    logger.filters[0].correlation_id = request.headers.get("X-Correlation-Id", uuid.uuid4())
    logger.filters[0].user_id = request.headers.get("X-User-Id", "-")

    logger.info("START REQUEST 1234")

    return response

错误点分析与解决方案 #

核心错误点 #

1. 未使用New Relic适配的日志Handler：普通的StreamHandler/FileHandler只会输出日志到本地或控制台，不会主动上报到New Relic平台。
2. 未绑定自定义Formatter到Handler：定义了_log_format但未设置到Handler的formatter属性，导致自定义字段无法被渲染。
3. Filter中自定义属性传递方式错误：newrelic.agent.add_custom_attributes不需要遍历record字段批量添加，且该方法是为APM事务添加属性，并非日志字段。

修正步骤与完整代码 #

1. 安装并初始化New Relic Agent

确保已安装newrelic包，并完成初始化（可通过newrelic-admin run-program启动应用，或代码中手动初始化）。

2. 配置New Relic日志Handler与自定义Formatter

使用newrelic.agent.NewRelicLogHandler来直接上报日志到New Relic，同时绑定自定义Formatter，确保自定义字段被正确渲染和传递。

3. 修正Filter与中间件逻辑

Filter只需确保record对象上的自定义字段被正确赋值，New Relic的Handler会自动提取这些字段作为日志的自定义属性。

import logging
import uuid
from fastapi import FastAPI, Request
import newrelic.agent

# 初始化New Relic Agent（若未通过newrelic-admin启动）
newrelic.agent.initialize('/path/to/newrelic.ini')

app = FastAPI()
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

# 自定义日志格式
_log_format = "%(asctime)s-%(correlation_id)s-%(levelname)s-%(message)s-%(http_method)s:%(http_endpoint)s-%(funcName)s-%(lineno)d-%(user_id)s-%(client_ip)s-%(user_agent)s"

# 1. 使用New Relic官方日志Handler
nr_handler = newrelic.agent.NewRelicLogHandler()
formatter = logging.Formatter(_log_format)
nr_handler.setFormatter(formatter)
logger.addHandler(nr_handler)

# 2. 自定义Filter，用于注入请求上下文字段
class RequestContextFilter(logging.Filter):
    def __init__(self):
        self.correlation_id = None
        self.user_id = None
        self.user_agent = None
        self.http_method = None
        self.http_endpoint = None
        self.client_ip = None

    def filter(self, record):
        # 将上下文字段注入日志record
        record.correlation_id = self.correlation_id
        record.user_id = self.user_id
        record.user_agent = self.user_agent
        record.http_method = self.http_method
        record.http_endpoint = self.http_endpoint
        record.client_ip = self.client_ip
        return True

# 添加Filter到logger
context_filter = RequestContextFilter()
logger.addFilter(context_filter)

# 3. FastAPI中间件，获取请求上下文并更新Filter字段
@app.middleware("http")
async def global_middleware(request: Request, call_next):
    # 从请求头提取或生成上下文字段
    context_filter.correlation_id = request.headers.get("X-Correlation-Id", str(uuid.uuid4()))
    context_filter.user_id = request.headers.get("X-User-Id", "-")
    context_filter.user_agent = request.headers.get("User-Agent", "-")
    context_filter.http_method = request.method
    context_filter.http_endpoint = request.url.path
    context_filter.client_ip = request.client.host

    # 记录测试日志
    logger.info("Request started")

    response = await call_next(request)
    return response

验证方式 #

启动应用后发起请求，查看New Relic日志面板：

• 日志内容会按自定义格式显示
• 自定义字段（如correlation_id、user_id）会作为日志的属性字段出现在面板中，可用于筛选和分析

内容的提问来源于stack exchange，提问作者tanvi