Outbox模式下分布式追踪上下文未正确传播求助

问题说明 #

基于Debezium实现Outbox功能，通过Docker Compose配置并启用OpenTelemetry。向Outbox表插入数据时，已将序列化的traceparent追踪上下文存入tracingspancontext字段，但Connector未复用该上下文，反而生成新的追踪信息添加到消息头，需要让消息头使用表中存储的追踪上下文。

相关配置 #

Dockerfile #

ARG DEBEZIUM_VERSION=2.1
FROM quay.io/debezium/connect:${DEBEZIUM_VERSION}
ENV KAFKA_CONNECT_JDBC_DIR=$KAFKA_CONNECT_PLUGINS_DIR/kafka-connect-jdbc \
    KAFKA_CONNECT_ES_DIR=$KAFKA_CONNECT_PLUGINS_DIR/kafka-connect-elasticsearch

ARG POSTGRES_VERSION=42.5.1
ARG KAFKA_JDBC_VERSION=5.3.2

# Deploy PostgreSQL JDBC Driver
RUN cd /kafka/libs && curl -sO https://jdbc.postgresql.org/download/postgresql-$POSTGRES_VERSION.jar

# Deploy Kafka Connect JDBC
RUN mkdir $KAFKA_CONNECT_JDBC_DIR && cd $KAFKA_CONNECT_JDBC_DIR &&\
    curl -sO https://packages.confluent.io/maven/io/confluent/kafka-connect-jdbc/$KAFKA_JDBC_VERSION/kafka-connect-jdbc-$KAFKA_JDBC_VERSION.jar

RUN mkdir /kafka/etc && cd /kafka/etc
COPY opentelemetry-javaagent.jar /kafka/etc

Docker Compose配置 #

connect:
  build:
    context: connector
  hostname: connect
  depends_on:
    kafka-broker-1:
      condition: service_healthy
    kafka-broker-2:
      condition: service_healthy
    kafka-broker-3:
      condition: service_healthy
  ports:
    - 8084:8083
  environment:
    - BOOTSTRAP_SERVERS=kafka-broker-1:9092,kafka-broker-2:9092,kafka-broker-3:9092
    - ENABLE_APICURIO_CONVERTERS=true
    - GROUP_ID=1
    - CONFIG_STORAGE_TOPIC=my_connect_configs
    - OFFSET_STORAGE_TOPIC=my_connect_offsets
    - STATUS_STORAGE_TOPIC=my_connect_statuses
    - ENABLE_OTEL=true
    - KAFKA_OPTS=-javaagent:/kafka/etc/opentelemetry-javaagent.jar
    - OTEL_SERVICE_NAME=sam-gmbh-kafka-connect
    - OTEL_TRACES_EXPORTER=otlp
    - OTEL_METRICS_EXPORTER=none
    - OTEL_LOGS_EXPORTER=none
    - OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318
    - OTEL_TRACES_SAMPLER=always_on
    - CONNECT_LOG4J_LOGGERS="DEBUG, stdout, appender"
    - OTEL_PROPAGATORS=tracecontext

tracingspancontext字段示例 #

#Wed Jun 26 16:29:09 CEST 2024
traceparent=00-a90cf8f35ba6dad17e8c9cdaafdbd0c1-3515e6f023a43bec-01

Connector配置 #

{
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "tasks.max": "1",
    "database.hostname": "postgres",
    "database.port": "5432",
    "database.user": "postgres",
    "database.password": "postgres",
    "database.dbname" : "postgres",
    "table.include.list": "product.outbox, order.outbox, payment.outbox",
    "transforms": "outbox",
    "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
    "transforms.outbox.route.by.field": "aggregate_type",
    "transforms.outbox.table.field.event.key": "aggregate_id",
    "transforms.outbox.table.field.event.payload": "payload",
    "transforms.outbox.table.expand.json.payload": "true",
    "transforms.outbox.route.topic.replacement": "${routedByValue}.events_ob",
    "transforms.outbox.tracing.span.context.field": "tracingspancontext",
    "transforms.outbox.tracing.operation.name": "debezium-read-sam",
    "transforms.outbox.tracing.with.context.field.only": "true",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "topic.prefix": "outbox",
    "value.converter.schemas.enable": false,
    "key.converter.schemas.enable": false,
    "slot.name": "outbox"
}

实际消息头追踪信息 #

{
    "traceparent": "00-d075554b98df047ba2795ca23e761969-77564be39bc4cc04-01",
    "id": "8bab4a69-976e-4842-8e5a-fce562847b38"
}

解决步骤 #

1. 清理tracingspancontext字段格式：当前字段包含注释行，Debezium的Outbox Transform可能无法正确解析带注释的内容。存储时只保留traceparent=00-a90cf8f35ba6dad17e8c9cdaafdbd0c1-3515e6f023a43bec-01这一行，移除注释。
2. 确认Debezium参数有效性：检查Debezium 2.1版本是否支持transforms.outbox.tracing.with.context.field.only参数，确保该配置确实能让Connector仅使用字段中的上下文，而非生成新的。
3. 调整OpenTelemetry代理配置：添加环境变量OTEL_INSTRUMENTATION_KAFKA_CONNECT_ENABLED=false，避免OTel代理自动生成新的追踪上下文干扰Connector的逻辑。
4. 查看DEBUG日志：通过已启用的DEBUG日志，搜索EventRouter或tracing相关条目，确认Connector是否读取到tracingspancontext字段的内容，以及解析过程中是否有异常。

内容的提问来源于stack exchange，提问作者Samraj Subramaniam