OpenTelemetry Collector配置Prometheus Exporter报错求助

我正尝试通过Flux部署OpenTelemetry Collector，配置Prometheus Exporter将指标发送至Prometheus以便在Grafana中查看，但配置过程遇到问题。我的OpenTelemetry Collector的HelmRelease配置如下：

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: "opentelemetry-collector"
  namespace: monitoring
spec:
  releaseName: opentelemetry-collector
  maxHistory: 3
  interval: 1m0s
  suspend: false
  chart:
    spec:
      chart: opentelemetry-collector
      version: "0.111.1"
      sourceRef:
        kind: HelmRepository
        name: open-telemetry-charts
        namespace: flux-system
  values:
    image:
      repository: otel/opentelemetry-collector-contrib
      tag: latest
    mode: deployment
    extraEnvs:
    - name: MY_POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
    - name: K8S_NODE_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    presets:
      hostMetrics:
        enabled: true
      kubeletMetrics:
        enabled: true
      kubernetesAttributes:
        enabled: true
      kubernetesEvents:
        enabled: true
    clusterRole:
      create: true
      name: "opentelemetry-collector-admin"
      rules:
      - verbs: ["*"]
        resources: ["*"]
        apiGroups: ["*"]
      - verbs: ["*"]
        nonResourceURLs: ["*"]
      clusterRoleBinding:
        name: "opentelemetry-collector-admin"
    serviceAccount:
      create: true
      name: "opentelemetry-collector-admin"
    config:
      receivers:
        kubeletstats:
          collection_interval: 10s
          auth_type: "serviceAccount"
          endpoint: https://${env:K8S_NODE_NAME}:10250
          insecure_skip_verify: true
          metric_groups:
            - container
            - pod
            - volume
            - node      
          extra_metadata_labels:
            - container.id
        k8s_cluster:
          collection_interval: 10s
          node_conditions_to_report: [Ready, MemoryPressure,DiskPressure,NetworkUnavailable]
          allocatable_types_to_report: [cpu, memory, storage, ephemeral-storage]
        k8s_events:
          auth_type : "serviceAccount"   
        otlp:
          protocols:
            grpc:
              endpoint: ${env:MY_POD_IP}:4317
            http:
              endpoint: ${env:MY_POD_IP}:4318
        prometheus:
          config:
            scrape_configs:
            - job_name: opentelemetry-collector
              scrape_interval: 10s
              static_configs:
              - targets:
                - ${env:MY_POD_IP}:8888
      exporters:
        debug:
          verbosity: detailed
        prometheus:
          endpoint: "kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090"
      processors:
        batch: {}
        k8sattributes:
          extract:
            metadata:
            - k8s.namespace.name
            - k8s.deployment.name
            - k8s.statefulset.name
            - k8s.daemonset.name
            - k8s.cronjob.name
            - k8s.job.name
            - k8s.node.name
            - k8s.pod.name
            - k8s.pod.uid
            - k8s.pod.start_time
          passthrough: false
          pod_association:
          - sources:
            - from: resource_attribute
              name: k8s.pod.ip
          - sources:
            - from: resource_attribute
              name: k8s.pod.uid
          - sources:
            - from: connection
        memory_limiter:
          check_interval: 5s
          limit_percentage: 80
          spike_limit_percentage: 25
      service:
        telemetry: 
          logs: 
            level: "debug" 
        pipelines:
          metrics:
            receivers:
              - otlp
              # - prometheus
              - k8s_cluster
              - kubeletstats
            processors:
              - batch
              - k8sattributes
              - memory_limiter
            exporters:
              - debug
              - prometheus
    ports:
      metrics:
        enabled: true
        containerPort: 8888
        servicePort: 8888
        protocol: TCP
      otlp-grpc:
        enabled: true
        containerPort: 4317
        servicePort: 4317
        protocol: TCP
      otlp-http:
        enabled: true
        containerPort: 4318
        servicePort: 4318
        protocol: TCP

    ingress:
      enabled: false

    serviceMonitor:
      enabled: true
      extraLabels:
        release: kube-prometheus-stack

运行时出现以下错误：

collector服务器运行结束并报错：无法启动管道：监听tcp:9091时绑定失败：无法分配请求的地址；关闭管道失败：无正在运行的监控例程；无正在运行的监控例程；无正在运行的监控例程（OpenTelemetry Collector Prometheus Exporter）

解决方案 #

核心原因 #

Prometheus Exporter默认会监听9091端口用于暴露自身指标，错误是因为该端口无法绑定——可能是Pod IP未就绪导致绑定特定IP失败、端口被占用，或者配置冲突。

具体修复步骤 #

1. 显式配置Prometheus Exporter的监听地址
   在exporters.prometheus下添加server.endpoint配置，指定绑定所有网卡（0.0.0.0），避免依赖Pod IP导致的绑定失败：

   exporters:
     debug:
       verbosity: detailed
     prometheus:
       endpoint: "kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090"
       server:
         endpoint: "0.0.0.0:9091"
   

2. 优化OTLP接收器的绑定配置
   当前OTLP接收器绑定了${env:MY_POD_IP}，Pod启动初期该IP可能未就绪，改为绑定所有网卡更稳定：

   otlp:
     protocols:
       grpc:
         endpoint: 0.0.0.0:4317
       http:
         endpoint: 0.0.0.0:4318
   

3. 检查端口冲突与预设配置

   • 确认Helm配置的ports段未占用9091（当前配置无冲突）；
   • 查看最终生成的Collector配置（执行kubectl get configmap -n monitoring opentelemetry-collector -o yaml），检查是否有预设（presets）自动添加的重复Prometheus Exporter配置，若有则调整或禁用冲突预设。

4. （可选）修改Exporter监听端口
   如果9091端口确实被占用，可以更换端口并在ports中添加对应配置：

   # 修改exporter配置
   prometheus:
     endpoint: "kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090"
     server:
       endpoint: "0.0.0.0:9092"
   # 添加端口定义
   ports:
     prometheus-exporter:
       enabled: true
       containerPort: 9092
       servicePort: 9092
       protocol: TCP
   

验证修复 #

修改配置后重新部署HelmRelease，查看Pod日志确认Collector正常启动：

kubectl logs -n monitoring deployment/opentelemetry-collector -f

内容的提问来源于stack exchange，提问作者naqi-eurus