Otel Collector向OpenSearch Data Prepper发送数据时出现连接错误

我正在测试OpenTelemetry，通过收集日志和追踪数据并发送至OpenSearch实例。参考OpenSearch官方文档搭建环境，但Otel Collector向Data Prepper发送数据时出现问题，无法定位原因。

部署环境（Docker Compose） #

name: log-aggregation

services:
  opensearch:
    ports:
      - "9200:9200"
      - "9600:9600"
    networks:
      - log-aggregation
    image: docker.io/opensearchproject/opensearch:latest
    environment:
      "discovery.type": 'single-node'
      "plugins.security.disabled": 'true'
      OPENSEARCH_INITIAL_ADMIN_PASSWORD: <password>

  opensearch-dashboard:
    ports:
      - "5601:5601"
    networks:
      - log-aggregation
    build: ./opensearch/
    image: opensearch-dashboards-no-security
    environment:
      OPENSEARCH_HOSTS: "http://opensearch:9200"
      "server.ssl.enabled": "false"

  data-prepper:
    ports:
      - "4900:4900"
      - "21890:21890"
    networks:
      - log-aggregation
    image: docker.io/opensearchproject/data-prepper:latest
    volumes:
      - '.\opensearch\pipelines.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml'
      - '.\opensearch\data-prepper-config.yaml:/usr/share/data-prepper/config/data-prepper-config.yaml'

  otel-collector:
    image: docker.io/otel/opentelemetry-collector:latest
    ports:
      - "4317:4317"
      - "4318:4318"
    networks:
      - log-aggregation
    command: ["--config=/etc/otel/config.yaml"]
    volumes:
      - "./opentelemetry/collector-config.yaml:/etc/otel/config.yaml"

networks:
  log-aggregation: {}

OpenSearch Dashboards 自定义镜像 Dockerfile #

FROM opensearchproject/opensearch-dashboards:latest
RUN /usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin remove securityDashboards       
COPY --chown=opensearch-dashboards:opensearch-dashboards opensearch_dashboards.yml /usr/share/opensearch-dashboards/config/

注：此为测试环境，不用于生产，OpenSearch和Dashboard可正常使用。

Otel Collector 配置 #

receivers:
  otlp:
    protocols:
      grpc:
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:

exporters:
  debug:
    verbosity: detailed
  otlp/data-prepper:
    endpoint: data-prepper:21890
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, otlp/data-prepper]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, otlp/data-prepper]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, otlp/data-prepper]
  telemetry:
    logs:
      level: "debug"

Data Prepper 主配置 #

ssl: false

Data Prepper 流水线配置 #

entry-pipeline:
  delay: "100"
  source:
    otel_trace_source:
      ssl: false
      authentication:
        unauthenticated:
  buffer:
    bounded_blocking:
      buffer_size: 10240
      batch_size: 160
  sink:
    - pipeline:
        name: "raw-pipeline"
    - pipeline:
        name: "service-map-pipeline"
raw-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  buffer:
    bounded_blocking:
      buffer_size: 10240
      batch_size: 160
  processor:
    - otel_trace_raw:
  sink:
    - opensearch:
        hosts: ["http://opensearch:9200"]
        insecure: true
        index_type: trace-analytics-raw
service-map-pipeline:
  delay: "100"
  source:
    pipeline:
      name: "entry-pipeline"
  buffer:
    bounded_blocking:
      buffer_size: 10240
      batch_size: 160
  processor:
    - service_map_stateful:
  sink:
    - opensearch:
        hosts: ["http://opensearch:9200"]
        insecure: true
        index_type: trace-analytics-service-map

已确认的正常状态 #

执行命令验证Data Prepper流水线创建正常：

curl.exe localhost:4900/list
{"pipelines":[{"name":"entry-pipeline"},{"name":"service-map-pipeline"},{"name":"raw-pipeline"}]}

问题现象 #

1. 向Otel Collector发送测试追踪数据时，仅返回partialSuccess响应：

curl.exe -i http://localhost:4318/v1/traces -H 'Content-Type: application/json' -d '@opentelemetry/span.json'  
HTTP/1.1 200 OK
Content-Type: application/json
Date: Sun, 21 Jul 2024 11:15:49 GMT
Content-Length: 21

{"partialSuccess":{}}

2. Otel Collector日志中出现连接错误：received goaway and there are no active streams，日志详情：

2024-07-21T11:06:49.213Z        info    zapgrpc/zapgrpc.go:176  [core] [Channel #3 SubChannel #6]Subchannel created     {"grpc_log": true}
2024-07-21T11:06:49.213Z       info    zapgrpc/zapgrpc.go:176  [core] [Channel #3]Channel Connectivity change to CONNECTING    {"grpc_log": true}
2024-07-21T11:06:49.213Z       info    zapgrpc/zapgrpc.go:176  [core] [Channel #3 SubChannel #6]Subchannel Connectivity change to CONNECTING   {"grpc_log": true}
2024-07-21T11:06:49.213Z       info    zapgrpc/zapgrpc.go:176  [core] [Channel #3 SubChannel #6]Subchannel picks a new address "10.89.1.14:21890" to connect   {"grpc_log": true}
2024-07-21T11:06:49.213Z       info    zapgrpc/zapgrpc.go:176  [pick-first-lb] [pick-first-lb 0xc00074b620] Received SubConn state update: 0xc00074b6b0, {ConnectivityState:CONNECTING ConnectionError:<nil>}  {"grpc_log": true}
2024-07-21T11:06:49.269Z       info    zapgrpc/zapgrpc.go:176  [core] [Channel #3 SubChannel #6]Subchannel Connectivity change to READY        {"grpc_log": true}
2024-07-21T11:06:49.269Z       info    zapgrpc/zapgrpc.go:176  [pick-first-lb] [pick-first-lb 0xc00074b620] Received SubConn state update: 0xc00074b6b0, {ConnectivityState:READY ConnectionError:<nil>}       {"grpc_log": true}
2024-07-21T11:06:49.269Z       info    zapgrpc/zapgrpc.go:176  [core] [Channel #3]Channel Connectivity change to READY {"grpc_log": true}
2024-07-21T11:07:04.348Z       info    zapgrpc/zapgrpc.go:176  [core] [Channel #3 SubChannel #6]Subchannel Connectivity change to IDLE {"grpc_log": true}
2024-07-21T11:07:04.348Z       info    zapgrpc/zapgrpc.go:176  [transport] [client-transport 0xc000ba2248] Closing: connection error: desc = "received goaway and there are no active streams" {"grpc_log": true}
2024-07-21T11:07:04.348Z       info    zapgrpc/zapgrpc.go:176  [transport] [client-transport 0xc000ba2248] loopyWriter exiting with error: connection error: desc = "received goaway and there are no active streams"  {"grpc_log": true}

注：已确认Data Prepper端口配置正确，改用其他端口（如21892）会出现connection refused错误。

内容的提问来源于stack exchange，提问作者Flowly