如何让OpenTelemetry Python SDK输出单行JSON格式日志

问题原因 #

默认自带的ConsoleLogExporter序列化日志时默认开启了JSON格式化缩进，所以输出是多行美化结构，不利于日志采集工具做单行关联解析，不符合Datadog这类APM平台的日志采集要求。

实现步骤 #

• 自定义单行日志导出器，序列化时关闭JSON缩进，直接输出单行字符串
  替换原有common.py中导入ConsoleLogExporter的逻辑，新增自定义导出器实现：

  import json
  import logging
  
  from opentelemetry.sdk._logs import (
      LogEmitterProvider,
      LoggingHandler,
      set_log_emitter_provider,
  )
  from opentelemetry.sdk._logs.export import BatchLogProcessor, LogExporter, LogExportResult
  from opentelemetry.sdk.resources import Resource
  from opentelemetry.semconv.resource import ResourceAttributes
  
  from local_machine_resource_detector import LocalMachineResourceDetector
  
  
  class SingleLineConsoleLogExporter(LogExporter):
      """自定义单行JSON日志导出器"""
      def export(self, batch):
          for log_record in batch:
              log_data = {
                  "body": log_record.body,
                  "severity_number": repr(log_record.severity_number),
                  "severity_text": log_record.severity_text,
                  "attributes": dict(log_record.attributes),
                  "timestamp": f"{log_record.timestamp.isoformat()}Z" if log_record.timestamp else None,
                  "trace_id": hex(log_record.trace_id),
                  "span_id": hex(log_record.span_id),
                  "trace_flags": log_record.trace_flags,
                  "resource": repr(log_record.resource)
              }
              # 无缩进序列化，输出单行JSON
              print(json.dumps(log_data, default=str))
          return LogExportResult.SUCCESS
  
      def shutdown(self):
          pass
  
  
  def configure_logger(name, version):
      local_resource = LocalMachineResourceDetector().detect()
      resource = local_resource.merge(
          Resource.create(
              {
                  ResourceAttributes.SERVICE_NAME: name,
                  ResourceAttributes.SERVICE_VERSION: version,
              }
          )
      )
      provider = LogEmitterProvider(resource=resource)
      set_log_emitter_provider(provider)
      # 替换为自定义的单行导出器
      exporter = SingleLineConsoleLogExporter()
      provider.add_log_processor(BatchLogProcessor(exporter))
      logger = logging.getLogger(name)
      logger.setLevel(logging.DEBUG)
      handler = LoggingHandler()
      logger.addHandler(handler)
      return logger
  

• common_runner.py不需要做任何修改，保持原有代码即可：

  from common import configure_logger
  
  logger = configure_logger("common_runner", "6.6.6")
  
  logger.debug(
      "common_runner.py module has been run",
      extra={
          "username": "Sid Vicous",
      },
  )
  

输出效果 #

运行后输出的日志即为期望的单行格式，可直接被Datadog等APM平台正常采集关联：

{"body": "common_runner.py module has been run", "severity_number": "<SeverityNumber.DEBUG: 5>", "severity_text": "DEBUG", "attributes": {"username": "Sid Vicous"}, "timestamp": "2022-07-13T14:40:08.595698Z", "trace_id": "0x00000000000000000000000000000000", "span_id": "0x0000000000000000", "trace_flags": 0, "resource": "BoundedAttributes({'telemetry.sdk.language': 'python', 'telemetry.sdk.name': 'opentelemetry', 'telemetry.sdk.version': '1.11.1', 'net.host.name': 'Doug.Ramirez', 'net.host.ip': '127.0.0.1', 'service.name': 'common_runner', 'service.version': '6.6.6'}, maxlen=None)"}

注：如果使用更高版本的OpenTelemetry Python SDK，_logs下的私有API命名可能有小幅调整，对应修改日志字段的取值逻辑即可，核心逻辑始终是序列化JSON时不设置缩进参数，直接输出单行字符串。

内容的提问来源于stack exchange，提问作者Doug Ramirez