如何过滤/转换OpenTelemetry中Prometheus源的NaN指标并存入InfluxDB

我在OpenTelemetry Collector中使用Prometheus接收器，收到来自HAProxy的含NaN（非数字）值的指标，需要在发送至不支持NaN的InfluxDB 3 Core后端前对其进行过滤/转换。

已尝试的配置方案 #

1. Prometheus接收器的metric_relabel_configs尝试 #

无法修改HAProxy源剔除NaN值，尝试在接收器配置中过滤，但不确定是否支持基于值的过滤：

receivers:
 prometheus:
   config:
     scrape_configs:
       - job_name: 'internalapigateway'
         scrape_interval: 15s
         metrics_path: /metrics
         scheme: http
         static_configs:
           - targets: ['internal_api_gateway:8405']
          metric_relabel_configs:
            - source_labels: [__value__]
              regex: NaN
              action: drop

2. Filter处理器尝试 #

测试了Filter处理器，但文档说明如果高层级遥测数据被丢弃，低层级条件不会被检查，导致所有Gauge指标被丢弃，且疑似NaN数据未进入过滤流程：

processors:
  filter/drop_nan_inf:
    error_mode: ignore
    metrics:
      metric:
        - type == METRIC_DATA_TYPE_GAUGE
      datapoint:
       # 不确定如何检查NaN，IsDouble()允许NaN
        - datapoint.value_double > -100000000 and datapoint.value_double < 100000000
       # - metric.type == METRIC_DATA_TYPE_GAUGE    # 此过滤不生效，需放在metric段
       # - IsString(datapoint.value_double)
       # - IsMatch(datapoint.value_double, "NaN")
       # - datapoint.value_double != datapoint.value_double  # NaN永不相等
       # - datapoint.value_double > -1.7976931348623157e+308 and datapoint.value_double < 1.7976931348623157e+308

3. Transform处理器尝试 #

测试Transform处理器时，修改指标描述生效，但修改NaN值无效果，疑似NaN数据未进入转换流程：

processors:
  transform/trans_nan_inf:
    error_mode: ignore
    metric_statements:
      - error_mode: ignore
        conditions:
          - metric.type == METRIC_DATA_TYPE_GAUGE
        statements:
          - set(datapoint.value_double, 123) # 测试值无效果
          - set(metric.description, "TEST")  # 此设置生效

核心问题 #

如何过滤/转换来自Prometheus源的含NaN值的指标，使其可存入InfluxDB后端？

OpenTelemetry示例数据日志 #

{
    "resource": {
        "service.instance.id": "0945af19-108b-4c69-9d7b-2c3f8d8dc5b0",
        "service.name": "otelcol-contrib",
        "service.version": "0.143.1"
    },
    "otelcol.component.id": "filter/drop_nan_inf",
    "otelcol.component.kind": "processor",
    "otelcol.pipeline.id": "metrics/influxdb",
    "otelcol.signal": "metrics",
    "condition": "metric.type == METRIC_DATA_TYPE_GAUGE",
    "match": true,
    "TransformContext": {
        "resource": {
            "attributes": {
                "service.name": "internalapigateway",
                "server.address": "testdocker.internal_api_gateway",
                "service.instance.id": "testdocker.internal_api_gateway:8405",
                "server.port": "8405",
                "url.scheme": "http"
            },
            "dropped_attribute_count": 0
        },
        "scope": {
            "attributes": {},
            "dropped_attribute_count": 0,
            "name": "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver",
            "version": "0.143.1"
        },
        "metric": {
            "description": "Limit on the number of connections in queue, for servers only (maxqueue argument)",
            "name": "haproxy_server_queue_limit",
            "unit": "",
            "type": "Gauge",
            "metadata": { "prometheus.type": "gauge" },
            "datapoints": [
                {
                    "attributes": {
                        "proxy": "http_back_testservice",
                        "server": "testservicesrv"
                    },
                    "exemplars": [],
                    "flags": 0,
                    "start_time_unix_nano": 0,
                    "time_unix_nano": 1767731654829000000,
                    "value_double": "NaN"
                },
                {
                    "attributes": {
                        "proxy": "tcp_back_outstation",
                        "server": "outstationsrv"
                    },
                    "exemplars": [],
                    "flags": 0,
                    "start_time_unix_nano": 0,
                    "time_unix_nano": 1767731654829000000,
                    "value_double": "NaN"
                }
            ]
        },
        "datapoint": {
            "attributes": {
                "proxy": "http_back_testservice",
                "server": "testservicesrv"
            },
            "exemplars": [],
            "flags": 0,
            "start_time_unix_nano": 0,
            "time_unix_nano": 1767731654829000000,
            "value_double": "NaN"
        },
        "cache": {}
    }
}

解决方案 #

方案1：修复Prometheus接收器的metric_relabel_configs配置 #

你的接收器配置存在缩进错误，修正后可在采集阶段直接丢弃NaN值，这是最高效的处理方式：

receivers:
 prometheus:
   config:
     scrape_configs:
       - job_name: 'internalapigateway'
         scrape_interval: 15s
         metrics_path: /metrics
         scheme: http
         static_configs:
           - targets: ['internal_api_gateway:8405']
         # 注意缩进需与static_configs同级
         metric_relabel_configs:
           - source_labels: [__value__]
             regex: 'NaN'
             action: drop

Prometheus会将__value__转为字符串，匹配NaN后直接丢弃对应数据点，从源头减少无效数据。

方案2：使用Filter处理器精准过滤NaN数据 #

如果接收器层面过滤不生效，可利用NaN的特性（x != x）在Filter处理器中匹配并丢弃：

processors:
  filter/drop_nan:
    error_mode: ignore
    metrics:
      datapoint:
        # 仅丢弃值为NaN的数据点，不影响其他指标
        - datapoint.value_double != datapoint.value_double
        action: exclude

无需额外匹配指标类型，只有Gauge类型的指标会有value_double字段，不会误删Counter等其他类型指标。

方案3：使用Transform处理器替换NaN为有效值 #

若不想丢弃数据，可将NaN替换为默认值（比如0）：

processors:
  transform/replace_nan:
    error_mode: ignore
    metric_statements:
      - conditions:
          - metric.type == METRIC_DATA_TYPE_GAUGE
          - datapoint.value_double != datapoint.value_double
        statements:
          - set(datapoint.value_double, 0)

该配置会针对所有Gauge类型的NaN数据点，将值替换为0，确保能正常写入InfluxDB。

注意事项 #

• 需将处理器添加到metrics流水线中，示例配置：

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [filter/drop_nan] # 或transform/replace_nan
      exporters: [influxdb]

• 确保使用的otelcol-contrib版本已启用Filter和Transform处理器（默认启用）。

内容的提问来源于stack exchange，提问作者paskalOn