OpenTelemetry Datadog Exporter发送重复Metric问题求助

在EKS集群的每个节点上以DaemonSet方式运行OpenTelemetry Collector并配置Datadog Exporter后，发现Metric的host标签同时标记了Pod名称和Metric报告来源的EC2实例ID，导致host标签值数量翻倍。直接使用Datadog Agent发送相同Metric时则显示正常。

异常情况（Datadog中host标签） #

理想状态（直接使用Datadog Agent） #

当前OpenTelemetry Collector ConfigMap配置 #

kind: ConfigMap
  metadata:
    annotations:
      kubernetes.io/description: Contains a CA bundle that can be used to verify the
        kube-apiserver when using internal endpoints such as the internal service
        IP or kubernetes.default.svc. No other usage is guaranteed across distributions
    uid: 22cd0904-b90c-4123-92b3-9775318e1638
- apiVersion: v1
  data:
    relay: |
      exporters:
        datadog:
          api:
            key: ${env:DD_API_KEY}
            site: datadoghq.com
          host_metadata:
            enabled: true
            hostname_source: first_resource
        debug: {}
      extensions:
        health_check: {}
        memory_ballast:
          size_in_percentage: 40
      processors:
        resourcedetection/eks:
          detectors: [env, eks]
          timeout: 2s
          override: true
        attributes/k8s_cluster_name:
          actions:
          - action: upsert
            key: k8s.cluster.name
            value: devel_plane_regional_us-west-2
        batch: {}
        k8sattributes:
          extract:
            labels:
            - from: pod
              key_regex: (.*)
              tag_name: $$1
            metadata:
            - k8s.namespace.name
            - k8s.deployment.name
            - k8s.statefulset.name
            - k8s.daemonset.name
            - k8s.cronjob.name
            - k8s.job.name
            - k8s.node.name
            - k8s.pod.name
            - k8s.pod.uid
            - k8s.pod.start_time
          filter:
            node_from_env_var: K8S_NODE_NAME
          passthrough: false
          pod_association:
          - sources:
            - from: resource_attribute
              name: k8s.pod.ip
          - sources:
            - from: resource_attribute
              name: k8s.pod.uid
          - sources:
            - from: connection
        memory_limiter:
          check_interval: 5s
          limit_percentage: 80
          spike_limit_percentage: 25
      receivers:
        hostmetrics:
          collection_interval: 10s
          root_path: /hostfs
          scrapers:
            cpu: null
            disk: null
            filesystem:
              exclude_fs_types:
                fs_types:
                - autofs
                - binfmt_misc
                - bpf
                - cgroup2
                - configfs
                - debugfs
                - devpts
                - devtmpfs
                - fusectl
                - hugetlbfs
                - iso9660
                - mqueue
                - nsfs
                - overlay
                - proc
                - procfs
                - pstore
                - rpc_pipefs
                - securityfs
                - selinuxfs
                - squashfs
                - sysfs
                - tracefs
                match_type: strict
              exclude_mount_points:
                match_type: regexp
                mount_points:
                - /dev/*
                - /proc/*
                - /sys/*
                - /run/k3s/containerd/*
                - /var/lib/docker/*
                - /var/lib/kubelet/*
                - /snap/*
            load: null
            memory: null
            network: null
        jaeger:
          protocols:
            grpc:
              endpoint: ${env:MY_POD_IP}:14250
            thrift_compact:
              endpoint: ${env:MY_POD_IP}:6831
            thrift_http:
              endpoint: ${env:MY_POD_IP}:14268
        otlp:
          protocols:
            grpc:
              endpoint: ${env:MY_POD_IP}:4317
            http:
              endpoint: ${env:MY_POD_IP}:4318
        prometheus:
          config:
            scrape_configs:
            - job_name: opentelemetry-collector_sp_regional_us-west-2
              scrape_interval: 10s
              static_configs:
              - targets:
                - ${env:MY_POD_IP}:8888
            - job_name: sp_control_plane_regional_us-west-2
              kubernetes_sd_configs:
              - role: pod
              relabel_configs:
              - action: keep
                regex: true
                source_labels:
                - __meta_kubernetes_pod_annotation_prometheus_io_scrape
              - action: keep
                regex: true
                source_labels:
                - __meta_kubernetes_pod_annotation_custom_telemetry
              - source_labels:
                - __meta_kubernetes_pod_container_port_name
                regex: metric
                action: keep
              - regex: __meta_kubernetes_pod_node_name
                action: labeldrop
              - action: replace
                regex: (.+)
                source_labels:
                - __meta_kubernetes_pod_annotation_prometheus_io_path
                target_label: __metrics_path__
              scrape_interval: 10s
        zipkin:
          endpoint: ${env:MY_POD_IP}:9411
      service:
        extensions:
        - health_check
        - memory_ballast
        pipelines:
          logs:
            exporters:
            - debug
            processors:
            - k8sattributes
            - memory_limiter
            - batch
            receivers:
            - otlp
          metrics:
            exporters:
            - otlp/newrelic
            - datadog
            - debug
            processors:
            - k8sattributes
            - memory_limiter
            - attributes/k8s_cluster_name
            - batch
            receivers:
            - prometheus
            - hostmetrics
          traces:
            exporters:
            - debug
            processors:
            - k8sattributes
            - memory_limiter
            - batch
            receivers:
            - otlp
            - jaeger
            - zipkin
        telemetry:
          metrics:
            address: ${env:MY_POD_IP}:8888
  kind: ConfigMap
  metadata:
    annotations:
      meta.helm.sh/release-name: otel-collector
      meta.helm.sh/release-namespace: otel-collector
    creationTimestamp: "2023-10-11T23:32:52Z"
    labels:
      app.kubernetes.io/instance: otel-collector
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: opentelemetry-collector
      app.kubernetes.io/version: 0.86.0
      helm.sh/chart: opentelemetry-collector-0.69.2
    name: otel-collector-opentelemetry-collector-agent
    namespace: otel-collector
kind: List
metadata: {}

调试日志关键行 #

2023-10-12T18:30:11.678Z        info    provider/provider.go:59 Resolved source {"kind": "exporter", "data_type": "metrics", "name": "datadog", "provider": "ec2", "source": {"Kind":"host","Identifier":"i-XXXXXXXXX"}}

已尝试的方案 #

• 修改Datadog Exporter的host_metadata.hostname_source为first_resource，无效果
• 在Prometheus scrape配置中添加__meta_kubernetes_pod_node_name的labeldrop规则，无效果

解决方案 #

1. 强制指定Datadog Exporter的hostname来源 #

禁用自动host元数据检测，直接从资源属性中读取Pod名称作为host标签：

exporters:
  datadog:
    api:
      key: ${env:DD_API_KEY}
      site: datadoghq.com
    host_metadata:
      enabled: false  # 关闭自动检测
    hostname: ${resource.attributes["k8s.pod.name"]}  # 手动指定为Pod名称

2. 确保resourcedetection处理器在metrics pipeline中生效 #

当前配置中resourcedetection/eks处理器未加入metrics pipeline，需要添加以正确识别EKS环境并补充k8s资源属性：

pipelines:
  metrics:
    exporters:
    - otlp/newrelic
    - datadog
    - debug
    processors:
    - resourcedetection/eks  # 新增该行
    - k8sattributes
    - memory_limiter
    - attributes/k8s_cluster_name
    - batch
    receivers:
    - prometheus
    - hostmetrics

3. 精简k8sattributes处理器的标签提取 #

避免冗余标签干扰，确保仅提取必要的Pod元数据：

k8sattributes:
  extract:
    metadata:
    - k8s.pod.name
    - k8s.node.name
    - k8s.namespace.name
    # 保留其他原有配置（filter、pod_association等）

完成以上修改后，重新部署OpenTelemetry Collector，Datadog中的host标签将仅显示Pod名称，与Datadog Agent的上报效果一致。

内容的提问来源于stack exchange，提问作者rushi47