Prometheus无法读取Docker中Otel Collector指标的配置问题

问题分析 #

你的配置框架逻辑没问题，但存在几个细节疏漏导致连接失败：

1. Prometheus可能在Otel Collector完全就绪前启动，初始连接超时
2. Otel Collector的OTLP Receiver仅开启HTTP协议，若Spring Boot默认用GRPC上报数据，会导致业务数据无法流转（额外影响链路）
3. 缺少服务健康检查机制，无法确保依赖服务就绪状态

修正后的完整配置 #

1. Docker Compose 文件

添加服务依赖和健康检查，确保Prometheus在Otel Collector就绪后启动，同时补充Spring Boot应用的配置：

version: '3.4'
networks:
  cart:
    driver: bridge

services:
  prometheus:
    build: prometheus
    ports:
      - "9090:9090"
    networks:
      - cart
    depends_on:
      otel-collector:
        condition: service_healthy

  otel-collector:
    build: otel-collector
    networks:
      - cart
    ports:
      - "1888:1888" # pprof extension
      - "8888:8888" # Collector自身监控指标
      - "8889:8889" # 转发给Prometheus的指标端口
      - "13133:13133" # 健康检查端口
      - "4317:4317" # OTLP gRPC接收端口
      - "4318:4318" # OTLP HTTP接收端口
      - "55679:55679" # zpages调试端口
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:13133/health"]
      interval: 10s
      timeout: 5s
      retries: 3

  spring-boot-app:
    build: ./spring-boot-app # 替换为你的Spring Boot项目路径
    networks:
      - cart
    ports:
      - "8080:8080"
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
      - OTEL_METRICS_EXPORTER=otlp
      - OTEL_SERVICE_NAME=spring-boot-cart-service
    depends_on:
      otel-collector:
        condition: service_healthy

2. Otel Collector 配置文件

同时开启OTLP的HTTP和GRPC协议，兼容Spring Boot默认上报方式：

receivers:
  otlp:
    protocols:
      http:
      grpc:

processors:
  batch:

exporters:
  prometheus:
      endpoint: 0.0.0.0:8889
      namespace: default

extensions:
  health_check:
  pprof:
  zpages:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

3. Prometheus 配置文件

添加超时配置，增强连接稳定性：

global:
  scrape_interval: 10s
  evaluation_interval: 10s
  
scrape_configs:
  - job_name: 'otel-collector'
    scrape_timeout: 5s
    static_configs:
      - targets: ["otel-collector:8889"]

4. Dockerfile 文件（无需修改）

你的Otel Collector和Prometheus的Dockerfile配置正确，保持原样即可。

验证步骤 #

1. 启动所有服务：docker-compose up -d
2. 进入Prometheus容器测试连通性：docker exec -it <prometheus-container-id> curl http://otel-collector:8889/metrics
   • 返回metrics数据则说明连接正常
3. 访问Prometheus UI（http://localhost:9090），在「Status > Targets」中查看otel-collector目标状态是否为UP

内容的提问来源于stack exchange，提问作者Kelly Goedert