使用Elastic APM时Otel Collector无法向Elasticsearch发送数据

问题背景 #

我是OpenTelemetry新手，采用YAML配置方式搭建链路监控栈，期望实现Otel Collector的数据转发至Elasticsearch，但遇到Elastic APM无法连接Elasticsearch的问题。

已验证正常的配置 #

• Otel Collector可正常接收数据
• Prometheus可正常接收数据

当前核心问题 #

• Elastic APM连接Elasticsearch时提示“连接拒绝”

技术栈 #

Otel Collector、Elastic APM、Elasticsearch、Prometheus

配置文件详情 #

docker-compose.yaml #

version: "3.7"
services:     
# APM Server   
  apm-server:
    image: docker.elastic.co/apm/apm-server:8.8.2
    volumes:
      - ./apm-server.docker.yml:/usr/share/apm-server/apm-server.yml
    cap_add: ["CHOWN", "DAC_OVERRIDE", "SETGID", "SETUID"]
    cap_drop: ["ALL"]
    ports:
      - "0.0.0.0:8200:8200"
    environment:
      host: 0.0.0.0:8200
    depends_on:
      - elasticsearch
    command: >
       apm-server -e
         -E apm-server.rum.enabled=true
         -E output.elasticsearch.hosts=['localhost:9200']
    networks:
      - icmnetworks  
# Elasticsearch     
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.8.2
    container_name: elasticsearch
    volumes:
      - ./elasticsearch.yaml:/usr/share/elasticsearch/config/elasticsearch.yml
    ports:
      - "0.0.0.0:9200:9200"
      - "0.0.0.0:9300:9300"
    environment:
      transport.host: 127.0.0.1
      ES_JAVA_OPTS: -Xms512m -Xmx512m
    mem_limit: 1073741824
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - icmnetworks      
  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml
    ports:
      - 9090:9090
    networks:
      - icmnetworks  
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    container_name: otel-collector
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./output:/etc/output:rw
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - 8888:8888   # Prometheus metrics exposed by the collector
      - 8889:8889   # Prometheus exporter metrics
      - 4317:4317   # OTLP gRPC receiver
      - 4318:4318   # OTLP http receiver
    depends_on:
      - prometheus
      - elasticsearch
    networks:
      - icmnetworks

networks:
  icmnetworks:
    driver: bridge

otel-collector-config.yaml #

receivers:
    otlp:
        protocols:
            grpc:
            http:

processors:
    batch:  

exporters:
    file:
        path: /etc/output/logs.json
    prometheus:
        endpoint: "0.0.0.0:9090"
    otlp/elastic:
        endpoint: "apm-server:8200"
        tls:
            insecure: true
    logging:
        loglevel: debug

service:
    pipelines:
        traces:
            receivers: [otlp]
            processors: [batch]
            exporters: [logging, otlp/elastic]
        metrics:
            receivers: [otlp]
            processors: [batch]
            exporters: [logging, prometheus]
        logs:
            receivers: [otlp]
            processors: []
            exporters: [logging, otlp/elastic] #, file

apm-server.yaml #

apm-server:
  host: "0.0.0.0:8200"
output.elasticsearch:
  hosts: ["localhost:9200", "0.0.0.0:9200"]
path.data: /usr/share/apm-server/data
path.logs: /var/log
logging.level: info
logging.to_syslog: true
logging.metrics.enabled: true
logging.files:
  path: /var/log/apm-server
  name: apm-server
  rotateeverybytes: 10485760 # = 10MB
  keepfiles: 7
  permissions: 0600
  interval: 0

elasticsearch.yaml #

cluster.name: elasticsearchcluster
node.name: elasticnode
node.attr.rack: r1
path.data: /usr/share/elasticsearch/data
path.logs: /usr/share/elasticsearch/logs
network.host: 0.0.0.0
http.port: 9200
discovery.type: single-node
xpack.security.enabled: false
action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*,+u*,+o*
cluster.routing.allocation.disk.threshold_enabled: false
cluster.routing.allocation.disk.watermark.low: 700mb
cluster.routing.allocation.disk.watermark.high: 600mb
cluster.routing.allocation.disk.watermark.flood_stage: 500mb
cluster.info.update.interval: 1m

prometheus.yaml #

scrape_configs:
  - job_name: 'otel-collector'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9090']

问题修复与优化方案 #

1. 解决Elastic APM连接Elasticsearch失败的核心问题 #

Docker Compose网络中，容器内的localhost指向自身容器，而非其他服务。你的APM Server配置中两处都用了localhost:9200，导致连接自身而非Elasticsearch容器。

修复步骤：

• 修改docker-compose中apm-server的command：

  command: >
     apm-server -e
       -E apm-server.rum.enabled=true
       -E output.elasticsearch.hosts=['elasticsearch:9200']
  

• 修改apm-server.yaml的输出配置：

  output.elasticsearch:
    hosts: ["elasticsearch:9200"]
  

2. 其他配置错误修复 #

(1) Prometheus采集目标错误

当前配置采集的是Prometheus自身的9090端口，需改为采集Otel Collector的metrics端口8888：

scrape_configs:
  - job_name: 'otel-collector'
    scrape_interval: 10s
    static_configs:
      - targets: ['otel-collector:8888']

(2) Otel Collector Prometheus exporter端口冲突

当前配置的endpoint: "0.0.0.0:9090"与Prometheus容器端口冲突，改为已映射的8889端口：

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"

(3) APM Server配置文件挂载不一致

docker-compose中挂载的是apm-server.docker.yml，但实际配置文件名是apm-server.yaml，修改挂载路径：

volumes:
  - ./apm-server.yaml:/usr/share/apm-server/apm-server.yml

(4) Elasticsearch transport.host冗余配置

elasticsearch容器中transport.host: 127.0.0.1会限制内部通信，直接删除该环境变量，使用yaml中的network.host: 0.0.0.0即可。

3. 验证方法 #

1. 重启所有容器：docker-compose down && docker-compose up -d
2. 查看APM Server日志：docker-compose logs apm-server，确认连接Elasticsearch成功
3. 查看Otel Collector日志：docker-compose logs otel-collector，确认数据转发至APM Server
4. 访问Elasticsearch索引列表：curl http://localhost:9200/_cat/indices?v，检查是否生成APM相关索引

内容的提问来源于stack exchange，提问作者Pratik Kumar