OpenTelemetry Collector通过OTLP导出至Jaeger失败求助

问题现象 #

多个客户端向OpenTelemetry Collector发送metrics、traces和logs，Collector能正常接收所有信号（可通过debug exporter输出到标准输出），但导出到Jaeger时失败，报错：
Permanent error: rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.metrics.v1.MetricsService

提供的配置及日志 #

Docker Compose配置 #

version: '3.9'
x-default-logging: &logging
  driver: "json-file"
  options:
    max-size: "5m"
    max-file: "2"

services:
  # ********************
  # Telemetry Components
  # ********************
  
  # Jaeger
  jaeger:
    image: jaegertracing/all-in-one:1.50
    container_name: jaeger
    deploy:
      resources:
        limits:
          memory: 500M
    restart: unless-stopped
    ports:
      - "16686:16686"                    # Jaeger UI
      - "4317"                           # OTLP gRPC default port
      - "4318"                           # OTLP HTTP default port (protobuf)
      - "14250"
    environment:
      - COLLECTOR_OTLP_ENABLED=true
    logging: *logging

  # OpenTelemetry Collector
  otelcol:
    image: otel/opentelemetry-collector-contrib:0.87
    container_name: otel-col
    deploy:
      resources:
        limits:
          memory: 525M
    restart: unless-stopped
    command: [ "--config=/etc/otelcol-config.yml" ]
    volumes:
      - ./otel/otelcol-config.yml:/etc/otelcol-config.yml
    ports:
      - "4317:4317"     # OTLP over gRPC receiver
      - "4318:4318"     # OTLP over HTTP receiver
      - "9464"          # Prometheus exporter
      - "8888"          # metrics endpoint
    depends_on:
      - jaeger
    logging: *logging

OpenTelemetry Collector配置（otelcol-config.yml） #

receivers:
  otlp:
    protocols:
      grpc:
      http:
        cors:
          allowed_origins:
            - "http://*"
            - "https://*"

exporters:
  debug:
    verbosity: detailed
  otlp:
    endpoint: "jaeger:4317"
    tls:
      insecure: true

processors:
  batch:

service:
  telemetry:
    logs:
      level: debug
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, otlp]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, otlp]

完整错误日志 #

2023-10-17T01:44:36.404Z    error   exporterhelper/retry_sender.go:145      Exporting failed. The error is not retryable. Dropping data.    {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "Permanent error: rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.metrics.v1.MetricsService", "dropped_items": 35}
otel-col  | go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
otel-col  |     go.opentelemetry.io/collector/exporter@v0.87.0/exporterhelper/retry_sender.go:145
otel-col  | go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
otel-col  |     go.opentelemetry.io/collector/exporter@v0.87.0/exporterhelper/metrics.go:176
otel-col  | go.opentelemetry.io/collector/exporter/exporterhelper.(*queueSender).start.func1
otel-col  |     go.opentelemetry.io/collector/exporter@v0.87.0/exporterhelper/queue_sender.go:126
otel-col  | go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).Start.func1
otel-col  |     go.opentelemetry.io/collector/exporter@v0.87.0/exporterhelper/internal/bounded_memory_queue.go:52

问题原因及解决办法 #

原因分析 #

Jaeger的all-in-one镜像（1.50版本）的OTLP接收器仅支持Traces信号，未实现Metrics和Logs的接收接口。报错信息中unknown service opentelemetry.proto.collector.metrics.v1.MetricsService直接说明Jaeger不支持Metrics服务的RPC调用。

解决步骤 #

1. 拆分Collector导出管道：

   • Traces信号继续通过OTLP exporter导出到Jaeger
   • Metrics和Logs信号需更换支持对应类型的后端（如Prometheus接收Metrics、Loki接收Logs），或直接移除对应管道的otlp exporter（若无需发送到Jaeger）

2. 修改Collector配置示例（保留Traces到Jaeger，Metrics/Logs仅用debug输出）：

exporters:
  debug:
    verbosity: detailed
  otlp:
    endpoint: "jaeger:4317"
    tls:
      insecure: true
  # 可选：添加prometheus exporter用于存储metrics
  # prometheus:
  #   endpoint: "0.0.0.0:9464"

processors:
  batch:

service:
  telemetry:
    logs:
      level: debug
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]  # 移除otlp exporter，或替换为prometheus等支持metrics的后端
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]  # 移除otlp exporter，或替换为loki等支持logs的后端

3. （可选）若需存储Metrics/Logs，可在Docker Compose中添加对应后端服务，并配置Collector的对应exporter。

内容的提问来源于stack exchange，提问作者xRed