如何清理OpenTelemetry JS Prometheus Exporter中的过期指标数据？

问题根源分析 #

1. 回调清理逻辑写反：原代码中cleanupMetrics方法的判断条件错误，导致未超时的设备指标被误清理，超时设备反而保留。
2. 位置指标建模不合理：将经纬度作为标签会触发时间序列爆炸，每次位置更新都会生成新指标，既浪费存储也不符合Prometheus最佳实践。
3. 离线状态判断逻辑缺失：没有专门的在线状态指标，无法通过指标本身判断设备是否离线。

一、修复回调清理逻辑Bug #

原代码的超时判断条件完全颠倒，修正后仅清理超过超时阈值的设备指标：

cleanupMetrics() {
  const timeoutMs = parseInt(process.env.METRICS_TIMEOUT_MINUTES) * 60 * 1000;
  for (const unit of Object.keys(this.callbacks)) {
    // 仅清理超时设备的指标回调
    if (Date.now() - this.callbacks[unit].timestamp >= timeoutMs) {
      for (const metric of this.reportingMetrics) {
        const callback = this.callbacks[unit][metric];
        if (callback) {
          this.metrics[metric].removeCallback(callback);
        }
      }
      delete this.callbacks[unit];
    }
  }
  setTimeout(this.cleanupMetrics.bind(this), 10000);
}

二、优化位置指标建模方式 #

放弃将经纬度作为标签的做法，拆分出独立的在线状态、纬度、经度指标，每个设备仅对应3条时间序列：

export class MetricsController {
  constructor(meter) {
    this.callbacks = {};
    // 调整指标集合，新增在线状态指标
    this.reportingMetrics = ["metric1", "metric2", "online_status", "latitude", "longitude"];
    this.metrics = this.createMetrics(meter);

    setTimeout(this.cleanupMetrics.bind(this), 10000);
  }

  createMetrics(meter) {
    const metrics = {};
    metrics.metric1 = meter.createObservableGauge("unit_metric1", {
      description: "Unit metric 1 value"
    });
    metrics.metric2 = meter.createObservableGauge("unit_metric2", {
      description: "Unit metric 2 value"
    });
    metrics.online_status = meter.createObservableGauge("unit_online_status", {
      description: "Unit online status (1=online, 0=offline)"
    });
    metrics.latitude = meter.createObservableGauge("unit_latitude", {
      description: "Unit current latitude",
      unit: "degrees"
    });
    metrics.longitude = meter.createObservableGauge("unit_longitude", {
      description: "Unit current longitude",
      unit: "degrees"
    });
    return metrics;
  }

  recordMetrics(incomingMetrics) {
    const unitId = incomingMetrics.unitId;
    if (!this.callbacks[unitId]) {
      this.callbacks[unitId] = {};
    }
    this.callbacks[unitId].timestamp = Date.now();

    // 先移除旧回调避免重复注册
    for (const metric of this.reportingMetrics) {
      if (this.callbacks[unitId][metric]) {
        this.metrics[metric].removeCallback(this.callbacks[unitId][metric]);
        delete this.callbacks[unitId][metric];
      }
    }

    // 注册新回调，标签仅保留设备ID
    this.callbacks[unitId].metric1 = (obs) => obs.observe(incomingMetrics.metric1, { unit_id: unitId });
    this.callbacks[unitId].metric2 = (obs) => obs.observe(incomingMetrics.metric2, { unit_id: unitId });
    this.callbacks[unitId].online_status = (obs) => obs.observe(1, { unit_id: unitId });
    this.callbacks[unitId].latitude = (obs) => obs.observe(incomingMetrics.latitude, { unit_id: unitId });
    this.callbacks[unitId].longitude = (obs) => obs.observe(incomingMetrics.longitude, { unit_id: unitId });

    for (const metric of this.reportingMetrics) {
      this.metrics[metric].addCallback(this.callbacks[unitId][metric]);
    }
  }

  cleanupMetrics() {
    const timeoutMs = parseInt(process.env.METRICS_TIMEOUT_MINUTES) * 60 * 1000;
    for (const unitId of Object.keys(this.callbacks)) {
      if (Date.now() - this.callbacks[unitId].timestamp >= timeoutMs) {
        for (const metric of this.reportingMetrics) {
          const callback = this.callbacks[unitId][metric];
          if (callback) {
            this.metrics[metric].removeCallback(callback);
          }
        }
        delete this.callbacks[unitId];
      }
    }
    setTimeout(this.cleanupMetrics.bind(this), 10000);
  }
}

三、Grafana Geomap面板配置调整 #

1. 数据源选择Prometheus
2. 查询模式选Table，执行PromQL：

   unit_online_status == 1
   

3. 位置设置选择Coordinates，分别关联unit_latitude和unit_longitude的指标值，用unit_id作为设备标识

这样在线设备会实时显示位置，离线设备因unit_online_status指标消失，会自动从地图上移除。

四、关于指标自动清理的说明 #

Prometheus是拉取式架构，Exporter本身不会主动清理指标。只要正确移除超时设备的回调，下次Prometheus抓取时就不会生成该设备的指标，Prometheus会根据自身的scrape_timeout和retention_policy自动清理过期的时间序列，无需额外实现抓取后清理逻辑。

内容的提问来源于stack exchange，提问作者Ethan Dutcher