配置OpenTelemetry Collector使用K8s节点主机名及解决图表无数据问题

背景 #

我将OpenTelemetry Collector（HostMetrics）以DaemonSet方式部署，节点指标发送至Prometheus/Mimir，通过Grafana看板查看数据。原本看板功能正常，但Host变量显示Collector Pod主机名，Pod回收后无效名称仍会保留。

尝试的配置修改 #

我添加处理器提取K8S_NODE_NAME作为service.hostname，修改片段如下：

resource/agent:
  attributes:
    - action: upsert
      key: service.namespace
      value: agent
    - action: upsert
      key: service.hostname
      value: ${env:K8S_NODE_NAME}
attributes/agent:
  actions:
    - key: service.namespace
      action: upsert
      value: agent
    - key: service.hostname
      action: upsert
      value: ${env:K8S_NODE_NAME}

修改后Host列表能显示节点名，但选择节点后图表无数据。

完整Collector配置 #

exporters:
  otlphttp/mimir:
    endpoint: http://mimir:9009/otlp

receivers:
  # otelcontribcol metrics + host metrics
  prometheus/agent:
    config:
      scrape_configs:
        - job_name: otel-collector
          scrape_interval: 10s
          static_configs:
            - targets: [0.0.0.0:8888]
  hostmetrics/agent:
    collection_interval: 10s
    scrapers:
      cpu:
        metrics:
          system.cpu.logical.count:
            enabled: true
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
          system.memory.limit:
            enabled: true
      load:
      disk:
      filesystem:
        metrics:
          system.filesystem.utilization:
            enabled: true
      network:
      paging:
      processes:
      process:
        mute_process_user_error: true
        metrics:
          process.cpu.utilization:
            enabled: true
          process.memory.utilization:
            enabled: true
          process.threads:
            enabled: true
          process.paging.faults:
            enabled: true

processors:
  batch/agent:
  resource/agent:
    attributes:
      - action: upsert
        key: service.namespace
        value: agent
  attributes/agent:
    actions:
      - key: service.namespace
        action: upsert
        value: agent
      - key: service.name
        action: upsert
        value: otel-collector
  resourcedetection/system:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]
  transform:
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["host.name"], resource.attributes["host.name"])
          - set(attributes["process.command"], resource.attributes["process.command"])
          - set(attributes["process.command_line"], resource.attributes["process.command_line"])
          - set(attributes["process.executable.name"], resource.attributes["process.executable.name"])
          - set(attributes["process.executable.path"], resource.attributes["process.executable.path"])
          - set(attributes["process.owner"], resource.attributes["process.owner"])
          - set(attributes["process.parent_pid"], resource.attributes["process.parent_pid"])
          - set(attributes["process.pid"], resource.attributes["process.pid"])

service:
  telemetry:
    metrics:
      level: detailed

  pipelines:
    metrics/agent:
      receivers: [prometheus/agent, hostmetrics/agent]
      processors: [attributes/agent, resourcedetection/system, transform, batch/agent]
      exporters: [otlphttp/mimir]

解决方向建议 #

• 修正处理器执行顺序：当前pipeline未加入新增的resource/agent处理器，导致service.hostname修改未生效。需将resource/agent加入处理器列表，且放在resourcedetection/system之前，避免系统检测器覆盖自定义主机名。修改后的pipeline处理器顺序建议：resource/agent, attributes/agent, resourcedetection/system, transform, batch/agent
• 验证环境变量注入：确保DaemonSet的Pod模板通过DownwardAPI注入节点名称到K8S_NODE_NAME环境变量，可进入Pod执行echo $K8S_NODE_NAME验证变量值是否正确
• 对齐看板关联字段：看板Host变量大概率关联host.name属性，而非service.hostname。可修改resource/agent处理器同时设置host.name，或调整transform处理器将service.hostname同步到host.name属性
• 检查指标标签：用PromQL查询system_cpu_logical_count等指标，确认返回的标签中包含正确的节点名称（如host.name或service_hostname）

内容的提问来源于stack exchange，提问作者jlim