Operator部署OpenTelemetry Collector出现指标重复问题求助

问题 #

环境配置 #

• 通过Otel Operator将Otel Collector部署为Kubernetes Deployment，共2个Pod
• 同一命名空间内的多个服务配置了自动插桩，遥测数据推送到Otel Collector Service
• 使用PodMonitor让Prometheus发现并采集Otel Pod的指标，两个Pod均已成为采集目标

问题现象 #

出现指标重复：同一服务的相同指标分别来自两个Otel Collector实例，指标值完全一致，仅Collector实例标识（如pod标签）不同。示例指标：

process_runtime_jvm_classes_loaded{app_kubernetes_io_component="opentelemetry-collector", app_kubernetes_io_instance="platform.otel", app_kubernetes_io_managed_by="opentelemetry-operator", app_kubernetes_io_name="otel-collector", app_kubernetes_io_part_of="opentelemetry", app_kubernetes_io_version="latest", container="otc-container", container_id="a8616d3c490b842c66601c64b2bfead4e84de8277c408c69c7c4efd30539142d", endpoint="prometheus", exported_job="cwb-be", host_arch="amd64", host_name="cwb-be-685c64f54c-zlzkb", instance="10.43.151.208:8889", job="platform/otel-collector-pod-monitor", k8s_container_name="cwb-be", k8s_deployment_name="cwb-be", k8s_namespace_name="platform", k8s_node_name="aks-aerasvc1-95583755-vmss000000", k8s_pod_name="cwb-be-685c64f54c-zlzkb", k8s_replicaset_name="cwb-be-685c64f54c", namespace="platform", os_description="Linux 5.15.0-1051-azure", os_type="linux", pod="otel-collector-5dccb6749f-x6xkj", pod_template_hash="5dccb6749f", process_pid="9", process_runtime_description="Amazon.com Inc. OpenJDK 64-Bit Server VM 25.372-b07", process_runtime_name="OpenJDK Runtime Environment", process_runtime_version="1.8.0_372-b07", service_name="cwb-be", service_version="2.7.0-main-b153", telemetry_auto_version="1.32.1", telemetry_sdk_language="java", telemetry_sdk_name="opentelemetry", telemetry_sdk_version="1.34.1"}
25691

重复指标（仅Collector实例不同）：

process_runtime_jvm_classes_loaded{app_kubernetes_io_component="opentelemetry-collector", app_kubernetes_io_instance="platform.otel", app_kubernetes_io_managed_by="opentelemetry-operator", app_kubernetes_io_name="otel-collector", app_kubernetes_io_part_of="opentelemetry", app_kubernetes_io_version="latest", container="otc-container", container_id="a8616d3c490b842c66601c64b2bfead4e84de8277c408c69c7c4efd30539142d", endpoint="prometheus", exported_job="cwb-be", host_arch="amd64", host_name="cwb-be-685c64f54c-zlzkb", instance="10.43.157.118:8889", job="platform/otel-collector-pod-monitor", k8s_container_name="cwb-be", k8s_deployment_name="cwb-be", k8s_namespace_name="platform", k8s_node_name="aks-aerasvc1-95583755-vmss000000", k8s_pod_name="cwb-be-685c64f54c-zlzkb", k8s_replicaset_name="cwb-be-685c64f54c", namespace="platform", os_description="Linux 5.15.0-1051-azure", os_type="linux", pod="otel-collector-5dccb6749f-7jhfg", pod_template_hash="5dccb6749f", process_pid="9", process_runtime_description="Amazon.com Inc. OpenJDK 64-Bit Server VM 25.372-b07", process_runtime_name="OpenJDK Runtime Environment", process_runtime_version="1.8.0_372-b07", service_name="cwb-be", service_version="2.7.0-main-b153", telemetry_auto_version="1.32.1", telemetry_sdk_language="java", telemetry_sdk_name="opentelemetry", telemetry_sdk_version="1.34.1"}
25691

核心疑问：

• 为何服务会将同一指标推送给两个Collector实例（均在K8s Service后端）
• 多实例部署是否有误
• 如何在非Prometheus/Grafana层面消除重复指标

解决方案 #

1. 明确重复原因 #

Kubernetes Service默认采用**轮询（Round Robin）**负载均衡策略，自动插桩的服务会将遥测数据发送到Service后端的所有Collector实例，导致同一份指标被多个Collector接收并暴露，最终被Prometheus采集两次。多实例部署本身没问题，这是正常的负载均衡行为，只需调整Collector或Service配置即可避免重复。

2. 可选解决方式 #

方式一：配置Service粘性会话，避免重复推送

修改Otel Collector Service的会话亲和性，让同一个服务Pod的遥测数据始终推送到同一个Collector实例：

apiVersion: v1
kind: Service
metadata:
  name: otel-collector
spec:
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 3600

优势：无需修改Collector配置，实现简单；劣势：可能导致Collector负载不均。

方式二：启用Collector跨实例去重

在Collector的pipeline中添加dedup处理器，借助共享存储（如Redis）实现跨实例的指标去重，确保同一份指标仅被暴露一次：

# Collector配置片段
processors:
  dedup:
    cache:
      redis:
        endpoint: redis:6379
        ttl: 60s

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [dedup, batch]
      exporters: [prometheus]

优势：保留负载均衡能力，避免重复指标；劣势：需要部署Redis作为共享存储，增加运维复杂度。

方式三：切换为Sidecar模式部署Collector

将Otel Collector以Sidecar模式注入到每个服务Pod中，每个服务Pod的遥测数据仅推送给本地的Sidecar Collector，从根源避免跨Collector的重复推送：

# 通过Otel Operator配置Sidecar示例
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: default
spec:
  exporter:
    endpoint: http://localhost:4317
  sidecar:
    image: otel/opentelemetry-collector-contrib:latest

优势：无需负载均衡配置，无重复推送问题；劣势：会增加每个服务Pod的资源消耗。

方式四：改用Remote Write统一推送指标

配置Collector的prometheusremotewrite exporter，将所有指标直接推送到Prometheus的远程写入端点，同时删除PodMonitor停止采集Collector的指标端口：

# Collector配置片段
exporters:
  prometheusremotewrite:
    endpoint: "http://prometheus:9090/api/v1/write"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheusremotewrite]

优势：利用Prometheus自身的时序去重逻辑消除重复，简化采集链路；劣势：依赖Prometheus的Remote Write能力。

3. 验证效果 #

调整配置后重启Otel Collector实例，使用PromQL查询验证指标是否重复：

count(process_runtime_jvm_classes_loaded{service_name="cwb-be"}) by (pod)

若每个服务实例的指标仅来自一个Collector，则说明配置生效。

内容的提问来源于stack exchange，提问作者Mircea