如何用Logback覆盖Google DataFlow日志并捕获Trace ID？

问题背景 #

我们在Google Cloud中部署了基于Apache Beam开发的DataFlow。当前DataFlow日志未包含追踪管道事务所需的事务ID，且Logback中配置的日志格式被Google Cloud忽略，需要实现在Google Cloud日志中捕获Trace ID。

现有配置代码 #

logback.xml #

<configuration >
<property name="projectId" value="${projectId:-${GOOGLE_CLOUD_PROJECT}}"/>
<appender name="CONSOLE_JSON" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
        <layout class="org.springframework.cloud.gcp.logging.StackdriverJsonLayout">
            <projectId>${projectId}</projectId>
            <includeTraceId>true</includeTraceId>
            <includeSpanId>true</includeSpanId>
            <includeLevel>true</includeLevel>
            <includeThreadName>true</includeThreadName>
            <includeMDC>true</includeMDC>
            <includeLoggerName>true</includeLoggerName>
            <includeFormattedMessage>true</includeFormattedMessage>
            <includeExceptionInMessage>true</includeExceptionInMessage>
            <includeContextName>true</includeContextName>
            <includeMessage>false</includeMessage>
            <includeException>false</includeException>
        </layout>
    </encoder>
</appender>

<root level="INFO">
    <appender-ref ref="CONSOLE_JSON"/>
</root>
</configuration>

Java代码 #

MDC.put("traceId", "12345");
log.info("Logging from test class");

当前Google Cloud日志输出 #

jsonPayload: {
  job: "2022-09-08_19_05_07-12432432432"
  logger: "TestLogger"
  message: "Logging from test class"
  stage: "A1"
  step: "Test Step"
  thread: "49"
  work: "3243243"
  worker: "test-worker"
}

解决步骤 #

1. 修正Trace ID的格式 #

Stackdriver要求Trace ID必须遵循projects/[PROJECT_ID]/traces/[TRACE_ID]的完整格式，否则无法被识别为可关联的追踪ID。修改Java代码：

String projectId = System.getenv("GOOGLE_CLOUD_PROJECT");
String traceId = "12345"; // 实际业务中建议使用UUID或Beam生成的原生Trace ID
MDC.put("traceId", String.format("projects/%s/traces/%s", projectId, traceId));
log.info("Logging from test class");

2. 确保自定义Logback配置在DataFlow中生效 #

DataFlow Worker默认会使用内置日志配置，需通过以下方式让自定义logback.xml优先加载：

• 将logback.xml打包到JAR包的src/main/resources目录下，确保Worker启动时能读取到。
• 提交DataFlow作业时，通过worker-log-level-overrides参数指定日志级别，同时确保自定义配置被加载：

gcloud dataflow jobs run YOUR_JOB_NAME \
  --gcs-location gs://YOUR_BUCKET/path/to/your/job.jar \
  --worker-log-level-overrides=ROOT=INFO,your.package=DEBUG \
  --project YOUR_PROJECT_ID

3. 结合Apache Beam的内置追踪机制 #

Apache Beam提供原生追踪支持，可自动生成并关联Trace ID，无需手动设置MDC：

// 初始化PipelineOptions时开启追踪
PipelineOptions options = PipelineOptionsFactory.create();
GcpOptions gcpOptions = options.as(GcpOptions.class);
gcpOptions.setProject("YOUR_PROJECT_ID");
TracingOptions tracingOptions = options.as(TracingOptions.class);
tracingOptions.setIsEnabled(true);

// 在DoFn中获取并关联当前Trace ID
public class YourDoFn extends DoFn<Input, Output> {
  @ProcessElement
  public void processElement(ProcessContext c) {
    Span span = Tracer.getCurrentSpan();
    String traceId = span.getContext().getTraceId();
    String projectId = c.getPipelineOptions().as(GcpOptions.class).getProject();
    MDC.put("traceId", String.format("projects/%s/traces/%s", projectId, traceId));
    log.info("Processing element with Trace ID: {}", traceId);
    // 业务逻辑处理
  }
}

4. 验证日志输出 #

修改后，Google Cloud日志的jsonPayload中应出现trace字段，且日志界面可直接点击Trace ID跳转到追踪详情页。若仍未出现，检查：

• 依赖是否包含spring-cloud-gcp-logging，版本是否与Spring Boot/Beam兼容。
• Worker是否正确加载自定义logback.xml，可在Worker日志中搜索logback关键词确认加载情况。
• MDC中的traceId是否正确设置，可添加日志打印MDC内容：log.info("MDC content: {}", MDC.getCopyOfContextMap());

内容的提问来源于stack exchange，提问作者Srinivas Reddy