Prometheus与Isilon Exporter拉取指标时出现重复数据错误

环境信息 #

• Prometheus版本：v2.46.0
• Isilon节点API版本：9.x
• Isilon Exporter：对应paychex开源实现

Exporter启动配置 #

prometheus-isilon-exporter -url http://10.120.71.9

Prometheus采集配置 #

- job_name: isilon-exporter-stats
  scrape_interval: 2m
  scrape_timeout: 1m
  static_configs:
  - targets:
    - 127.0.0.1:9437

报错信息 #

直接访问http://prometheus01:9437/metrics时出现大量重复指标错误，示例如下：

An error has occurred while serving metrics:

3736 error(s) occurred:
* collected metric "emcisi_cluster_hard_quota" { label:<name:"clustername" value:"itisimdcp12" > label:<name:"path" value:"/ifs/mdcp12/o2/data2/bch/surg/vakili" > gauge:<value:0 > } was collected before with the same name and label values
* collected metric "emcisi_cluster_advisory_quota" { label:<name:"clustername" value:"itisimdcp12" > label:<name:"path" value:"/ifs/mdcp12/o2/data2/bch/surg/vakili" > gauge:<value:0 > } was collected before with the same name and label values
* collected metric "emcisi_cluster_logical_used" { label:<name:"clustername" value:"itisimdcp12" > label:<name:"path" value:"/ifs/mdcp12/o2/data2/bch/surg/vakili" > gauge:<value:158 > } was collected before with the same name and label values
* collected metric "emcisi_cluster_physical_used" { label:<name:"clustername" value:"itisimdcp12" > label:<name:"path" value:"/ifs/mdcp12/o2/data2/bch/surg/vakili" > gauge:<value:32768 > } was collected before with the same name and label values
...

已尝试操作 #

• 延长Prometheus的scrape_interval，问题未解决
• 切换Isilon Exporter多模式运行，报错依旧

解决方法 #

1. 排查Isilon API返回数据
   直接调用Isilon的配额API（如/platform/1/quota/quotas），检查返回的JSON中是否存在相同clustername和path的重复条目——Isilon 9.x API可能存在此类数据重复问题。

2. 给Exporter添加去重逻辑
   修改Exporter代码，在采集指标环节以clustername+path作为唯一标识，跳过已采集的重复条目：

• 遍历API返回数据时，维护一个去重字典
• 遇到已存在的标识键，直接跳过指标生成

3. 临时用Prometheus relabel规则过滤
   如果暂时无法修改Exporter，可在采集配置中添加规则保留唯一指标实例：

- job_name: isilon-exporter-stats
  scrape_interval: 2m
  scrape_timeout: 1m
  static_configs:
  - targets:
    - 127.0.0.1:9437
  metric_relabel_configs:
  - source_labels: [__name__, clustername, path]
    action: hashmod
    modulus: 1
    target_label: __tmp_hash
  - source_labels: [__tmp_hash]
    action: keep
    regex: 0

注：此为临时 workaround，仅能过滤重复，无法解决根源问题。

4. 验证Exporter版本兼容性
   拉取Exporter项目最新代码编译测试，确认是否已有针对Isilon 9.x API的重复数据修复。

内容的提问来源于stack exchange，提问作者Johnny Simp