OpenSearch Trace Analytics：DataPrepper传输Trace的延迟问题求助

架构路径：应用Trace → OTEL Collector → DataPrepper → OpenSearch

环境配置 #

• OpenSearch和Dashboards v2.12.0
• DataPrepper v2.7 & 2.8
• OTEL Collector v0.102.1
• 操作系统：Oracle Linux 8

问题描述 #

包含多Span的Trace从生成到能在OpenSearch Dashboards中查看存在5-10分钟的延迟。经排查，OTEL Collector接收Trace后几乎立即转发至DataPrepper，但DataPrepper处理多Span Trace并发送至OpenSearch耗时过长。测试单条约150KB的Trace，调整堆内存、worker数量、缓冲区大小后问题仍未解决，且DataPrepper日志（已开启debug）无警告/错误信息，需协助排查。

OTEL Collector配置 #

extensions:
  health_check:
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  prometheus:
    config:
      scrape_configs:
      - job_name: 'otel-collector'
        scrape_interval: 10s
        static_configs:
        - targets: ['0.0.0.0:8888']
processors:
  batch:
    send_batch_size: 500
    timeout: 1s
exporters:
  debug:
    verbosity: detailed
    sampling_initial: 5
    sampling_thereafter: 200
  otlp/data-prepper:
    endpoint: '<dataprepper-ip>:21890'
    tls:
      insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug,otlp/data-prepper]

DataPrepper配置 #

entry-pipeline:
  workers: 4    
  delay: "5"
  source:
    otel_trace_source:
      ssl: false
  buffer:
    bounded_blocking:
      buffer_size: 25600
      batch_size: 500
  sink:
    - pipeline:
        name: "raw-trace-pipeline"
    - pipeline:
        name: "service-map-pipeline"
raw-trace-pipeline:
  workers: 8      
  source:
    pipeline:
      name: "entry-pipeline"
  buffer:
    bounded_blocking:
      buffer_size: 25600
      batch_size: 500
  processor:
    - otel_trace_raw:
  sink:
    - opensearch:
        hosts: ["https://localhost:9200"]
        insecure: true
        username: admin
        password: admin
        index_type: trace-analytics-raw     
service-map-pipeline:
  workers: 8
  delay: "5"
  source:
    pipeline:
      name: "entry-pipeline"
  buffer:
    bounded_blocking:
      buffer_size: 25600
      batch_size: 500
  processor:
    - service_map_stateful:
  sink:
    - opensearch:
        hosts: ["https://localhost:9200"]
        insecure: true
        username: admin
        password: admin
        index_type: trace-analytics-service-map

内容的提问来源于stack exchange，提问作者AmythD