在OpenShift中使用OpenTelemetry Collector Kafka导出器遇配置问题

我在OpenShift的不同命名空间分别部署了Kafka集群和OpenTelemetry Collector，使用以下配置YAML：

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: demo-otlcol-otlp
  namespace: otlcol-demo
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      batch:

    exporters:
      logging:
        loglevel: info

      kafka:
        brokers: 
          - dev-kafka-kafka-brokers.kafka.svc.cluster.local:9092
        protocol_version: 3.4.0

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [logging,kafka]
  mode: daemonset
  resources: {}
  targetAllocator: {}

运行时出现错误：

Error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:
* error decoding 'exporters': unknown type: "kafka" for id: "kafka" (valid values: [logging otlp otlphttp jaeger])
2023/05/31 09:24:46 collector server run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:
* error decoding 'exporters': unknown type: "kafka" for id: "kafka" (valid values: [logging otlp otlphttp jaeger])

需要指导如何正确在OpenTelemetry导出器中使用Kafka。

1. 更换包含Kafka Exporter的Collector镜像 #

默认的OpenTelemetry Collector镜像（otel/opentelemetry-collector）仅包含核心组件，Kafka Exporter属于contrib扩展组件，不在默认镜像中。你需要使用包含contrib组件的镜像：

• 官方镜像：otel/opentelemetry-collector-contrib:<版本号>
• OpenShift环境也可使用Red Hat提供的对应镜像，比如registry.redhat.io/openshift-otel/opentelemetry-collector-contrib-rhel8:<版本号>

修改你的OpenTelemetryCollector CR，添加image字段指定镜像：

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: demo-otlcol-otlp
  namespace: otlcol-demo
spec:
  # 指定包含Kafka Exporter的镜像
  image: otel/opentelemetry-collector-contrib:0.85.0
  config: |
    # 原配置内容保持不变
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      batch:

    exporters:
      logging:
        loglevel: info

      kafka:
        brokers: 
          - dev-kafka-kafka-brokers.kafka.svc.cluster.local:9092
        protocol_version: 3.4.0

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [logging,kafka]
  mode: daemonset
  resources: {}
  targetAllocator: {}

注意：请选择与你的Kafka版本兼容的Collector版本，避免协议不匹配问题。

2. 验证跨命名空间访问权限 #

确保OTel Collector所在的otlcol-demo命名空间能够访问Kafka所在的kafka命名空间：

• 检查OpenShift网络策略，允许otlcol-demo命名空间的Pod访问kafka命名空间的9092端口（Kafka broker端口）
• 测试连通性：在Collector Pod中执行nc -zv dev-kafka-kafka-brokers.kafka.svc.cluster.local 9092验证是否能连接到Kafka

3. 可选：完善Kafka Exporter配置 #

如果你的Kafka集群启用了认证（比如SASL/PLAIN、SSL），需要在Kafka Exporter配置中添加对应参数，示例：

exporters:
  kafka:
    brokers: 
      - dev-kafka-kafka-brokers.kafka.svc.cluster.local:9092
    protocol_version: 3.4.0
    sasl:
      username: "your-username"
      password: "your-password"
      mechanism: "PLAIN"
    tls:
      insecure: false

内容的提问来源于stack exchange，提问作者Mugi Coder