k6无法通过OpenTelemetry Collector写入Mimir/Tempo求助

报错信息 #

INFO[0030] 2025/01/15 22:47:29 failed to upload metrics: context deadline exceeded: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF"

已尝试的配置 #

OpenTelemetry Collector配置 #

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  # 指标：使用Prometheus Remote Write导出器发送数据到Grafana Mimir
  prometheusremotewrite:
    endpoint: "http://mimir:9009/api/v1/remote_write"
    # Mimir端点和认证信息需根据实际情况调整

  # 追踪：使用OTLP导出器发送追踪数据到Tempo
  otlp:
    endpoint: "tempo:4317"
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheusremotewrite]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

docker-compose配置 #

networks:
  observability:
    driver: bridge

services:
  # OpenTelemetry Collector (Contrib) 
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    container_name: otel-collector
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro
    ports:
      - "12345:12345"
      # - "4418:4318"
    networks:
      - observability

  # Grafana Tempo 追踪存储
  tempo:
    image: grafana/tempo:latest
    container_name: tempo
    command: ["-config.file=/etc/tempo/tempo.yaml"]
    ports:
      - "3200:3200"   # 查询用HTTP API
      - "4317:4317"   # OTLP 数据接入
      - "4318:4318"
    volumes:
      - ./tempo.yaml:/etc/tempo/tempo.yaml:ro
      - ./tempo-data:/var/tempo
    networks:
      - observability

  # Grafana Mimir 指标存储
  mimir:
    image: grafana/mimir:latest
    container_name: mimir
    # 实际配置取决于你的Mimir设置；这里假设它在9009端口接收remote write请求
    ports:
      - "9009:9009"
    volumes:
      - ./mimir-config.yaml:/etc/mimir/mimir.yaml:ro
      - ./mimir-data:/data
    networks:
      - observability

  # Grafana 可视化
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_DISABLE_LOGIN_FORM=true
      - GF_FEATURE_TOGGLES_ENABLE=traceqlEditor metricsSummary
      - GF_INSTALL_PLUGINS=https://storage.googleapis.com/integration-artifacts/grafana-exploretraces-app/grafana-exploretraces-app-latest.zip;grafana-traces-app      
    volumes:
      - ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
      # - ./grafana-dashboards.yaml:/etc/grafana/provisioning/dashboards/dashboards.yaml:ro
    ports:
      - "3000:3000"
    depends_on:
      - mimir
      - tempo
    networks:
      - observability

k6运行命令 #

K6_OTEL_HTTP_EXPORTER_INSECURE=true K6_OTEL_HTTP_EXPORTER_ENDPOINT=http://localhost:12345 K6_OTEL_METRIC_PREFIX=k6_ k6 run --out experimental-opentelemetry ./demo.js

问题排查与解决步骤 #

1. 修复端口映射不匹配问题 #

你的OpenTelemetry Collector启用了OTLP HTTP协议，但docker-compose中仅映射了未使用的12345端口，默认OTLP HTTP监听端口是4318，导致k6无法连接到Collector。

修改方案：

• 更新docker-compose中otel-collector的端口配置：

  ports:
    - "4317:4317"  # OTLP gRPC端口
    - "4318:4318"  # OTLP HTTP端口（k6需要连接这个）
  

• 调整k6运行命令，指向正确的端口：

  K6_OTEL_HTTP_EXPORTER_INSECURE=true K6_OTEL_HTTP_EXPORTER_ENDPOINT=http://localhost:4318 K6_OTEL_METRIC_PREFIX=k6_ k6 run --out experimental-opentelemetry ./demo.js
  

2. 修正Mimir Remote Write端点路径 #

Grafana Mimir默认的指标推送端点是/api/v1/push，而非Prometheus默认的/api/v1/remote_write，需要调整Collector的exporter配置：

prometheusremotewrite:
  endpoint: "http://mimir:9009/api/v1/push"

3. 验证容器间网络连通性 #

进入otel-collector容器，测试与Mimir、Tempo的连通性：

# 测试Mimir HTTP连接
docker exec -it otel-collector curl http://mimir:9009/api/v1/push -X POST
# 测试Tempo gRPC连接
docker exec -it otel-collector grpcurl -plaintext tempo:4317 list

如果无法连通，确认所有服务都加入了observability网络。

4. 调整Collector批处理超时（可选） #

若数据量较大，默认批处理设置可能导致超时，可增加超时时间：

processors:
  batch:
    timeout: 10s
    send_batch_size: 1000

5. 查看Collector日志排查细节 #

通过日志获取更详细的错误信息：

docker logs otel-collector

内容的提问来源于stack exchange，提问作者Chris G.