EKS集群中OpenTelemetry Collector无法连接New Relic问题排查

完整修正后的配置 #

ConfigMap #

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-configmap
  namespace: observability
data:
  collector.yaml: |
    receivers:
      otlp:
        protocols: 
          grpc:
          http:
    processors:
      batch:
        send_batch_size: 5000
        timeout: 10s
    exporters:
      otlp:
        endpoint: "otlp.nr-data.net:4317"
        tls:
          insecure: false
        headers:
          "api-key": "${NEW_RELIC_API_KEY}"
    service:
      telemetry:
        logs:
          level: "debug"
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlp]
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlp]
        logs:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlp]

Deployment #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector-deployment
  namespace: observability
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: otel-collector
  template:
    metadata:
      labels:
        app.kubernetes.io/name: otel-collector
    spec:
      containers:
        - name: otel-collector
          args:
            - sh
            - -c
            - "envsubst < /conf/collector.yaml > /tmp/collector.yaml && otelcol --config=/tmp/collector.yaml"
          image: otel/opentelemetry-collector-contrib:0.93.0
          env:
            - name: NEW_RELIC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: newrelic-secret
                  key: api-key
                  optional: false
          volumeMounts:
            - name: otel-collector-configmap
              mountPath: /conf
          resources:
            limits:
              cpu: 500m
              memory: 512Mi
      volumes:
        - name: otel-collector-configmap
          configMap:
            name: otel-collector-configmap 
            items:
              - key: collector.yaml
                path: collector.yaml

额外排查方向 #

1. 检查集群网络策略：即使开放了安全组，K8s集群的NetworkPolicy可能限制了Pod的出站流量，需要确认是否允许Pod访问otlp.nr-data.net:4317或4318。
2. 验证API密钥权限：确保你的New Relic API密钥是Ingest类型，拥有发送遥测数据的权限，且关联的账号正确。
3. 切换到OTLP HTTP协议：如果gRPC始终无法连通，可以尝试用OTLP HTTP exporter，配置示例：

exporters:
  otlphttp:
    endpoint: "https://otlp.nr-data.net:4318/v1/logs"
    headers:
      "api-key": "${NEW_RELIC_API_KEY}"

记得把pipelines里的exporters改成[otlphttp]。

内容的提问来源于stack exchange，提问作者bmoody