Jaeger+OpenTelemetry上报Trace数据至OpenSearch故障排查

三个报错为连锁触发，核心故障按影响优先级排序：

• 第一优先级：Data Prepper的管道配置YAML格式错误，entry-pipeline没有作为顶级键定义，服务启动时识别不到任何有效管道直接退出，是所有故障的源头
• 第二优先级：Data Prepper进程退出后21890端口无监听，导致OpenTelemetry Collector（以下简称OTel Collector）发起连接时被拒绝
• 第三优先级：手动硬编码OTel Collector容器IP的方式不可靠，加上服务启动顺序无校验，Jaeger Agent上报时网络不通触发i/o超时；最初遇到的DNS解析失败是服务启动时差导致的临时问题，无需硬编码IP解决。

1. 修复Data Prepper核心配置（最高优先级） #

你挂载的trace_analytics_no_ssl.yml存在两个致命问题：一是entry-pipeline未作为顶级YAML键声明，Data Prepper完全识别不到管道配置；二是OpenSearch地址写了localhost，容器内localhost指向Data Prepper自身，无法访问同网络下的OpenSearch服务。
将配置文件替换为以下正确内容：

entry-pipeline:
  delay: "100"
  source:
    otel_trace_source:
      ssl: false
  sink:
    - pipeline:
        name: "raw-pipeline"
    - pipeline:
        name: "service-map-pipeline"
raw-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  prepper:
    - otel_trace_raw_prepper:
  sink:
    - opensearch:
        hosts: [ "http://opensearch:9200" ]
        cert: "/usr/share/data-prepper/root-ca.pem"
        username: "admin"
        password: "admin"
        trace_analytics_raw: true
service-map-pipeline:
  delay: "100"
  source:
    pipeline:
      name: "entry-pipeline"
  prepper:
    - service_map_stateful:
  sink:
    - opensearch:
        hosts: ["http://opensearch:9200"]
        cert: "/usr/share/data-prepper/root-ca.pem"
        username: "admin"
        password: "admin"
        trace_analytics_service_map: true

同时在docker-compose的data-prepper服务配置中补充健康检查段，确保依赖它的服务等它完全启动再拉起：

healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:4900/health"]
      interval: 5s
      timeout: 2s
      retries: 10

2. 修复OpenTelemetry Collector配置 #

你现有OTel Collector的管道配置无语法问题，仅需调整docker-compose中的服务依赖规则、补全Jaeger gRPC接收端口的声明，修改后的配置段如下：

otel-collector:
    container_name: otel-collector
    image: otel/opentelemetry-collector:0.54.0
    command: [ "--config=/etc/otel-collector-config.yml" ]
    working_dir: "/project"
    volumes:
      - ${PWD}/:/project
      - ./otel-collector-config.yml:/etc/otel-collector-config.yml
      - ./data-prepper/examples/demo/demo-data-prepper.crt:/etc/demo-data-prepper.crt
    ports:
      - "4317:4317"
      - "14250:14250"
    depends_on:
      data-prepper:
        condition: service_healthy
      opensearch:
        condition: service_healthy
    networks:
      - our-network

补充说明：同docker自定义网络下容器间端口默认互通，暴露14250是为了方便外部排障，同时避免网络策略拦截Jaeger gRPC上报流量。

3. 修复Jaeger Agent配置 #

删掉硬编码的OTel Collector固定容器IP，改回docker服务名解析，同时增加失败重启策略解决启动时差导致的临时DNS报错，修改后的配置段如下：

jaeger-agent:
    container_name: jaeger-agent
    image: jaegertracing/jaeger-agent:latest
    command: [ "--reporter.grpc.host-port=otel-collector:14250" ]
    ports:
      - "5775:5775/udp"
      - "6831:6831/udp"
      - "6832:6832/udp"
      - "5778:5778/tcp"
    depends_on:
      otel-collector:
        condition: service_started
    networks:
      - our-network
    restart: on-failure

4. 补充OpenSearch健康检查 #

在docker-compose的opensearch服务段补充健康检查，避免上游服务启动时OpenSearch还未完成初始化：

healthcheck:
      test: ["CMD", "curl", "-f", "https://localhost:9200/_cluster/health", "-k", "-u", "admin:admin"]
      interval: 10s
      timeout: 5s
      retries: 20

1. 清理旧环境：执行docker-compose down -v删除所有异常退出的旧容器和挂载缓存
2. 启动全栈服务：执行docker-compose up -d拉起所有组件
3. 按顺序校验日志：
   • 查看Data Prepper日志，确认无No valid pipeline报错，出现服务启动成功字样即为正常
   • 查看OTel Collector日志，确认无连接21890被拒绝的报错
   • 查看Jaeger Agent日志，确认无连接14250超时的报错
4. 触发WordPress侧的页面访问请求，进入OpenSearch Trace Analytics页面确认链路数据正常写入。

内容的提问来源于stack exchange，提问作者issrae