Prometheus多主机多metric_path配置异常：自动追加/metric

我有数百台服务器，每台包含数十个metrics抓取路径。目前Prometheus可以抓取所有主机，但会自动在每个主机后追加/metrics。当前Prometheus配置和target.json内容如下，期望实现针对每个主机的指定路径进行抓取，而非默认的/metrics。

当前Prometheus配置 #

global:
  scrape_interval: 30s

scrape_configs:
  - job_name: 'file_sd_targets'  
    metrics_path: '/configuration/1'
    
    file_sd_configs:
      - files:
        - /etc/prometheus/targets.json
    relabel_configs:
      - source_labels: [__address__]
        regex: '([^:]+):.*'
        target_label: instance
        replacement: '$1'

当前target.json内容 #

[ 
  { "target": "1.2.3.4:10091", "metrics_paths": ["/configuration/1", "/analytics/1"] }, 
  { "target": "1.2.3.5:10091", "metrics_paths": ["/configuration/1", "/analytics/1"] }, 
  { "target": "1.2.3.6:10091", "metrics_paths": ["/configuration/1", "/analytics/1"] } 
]

期望的抓取路径 #

http://1.2.3.4:10091/configuration/1
http://1.2.3.4:10091/analytics/1
http://1.2.3.5:10091/configuration/1
http://1.2.3.5:10091/analytics/1
http://1.2.3.6:10091/configuration/1
http://1.2.3.6:10091/analytics/1

1. 修改target.json结构 #

Prometheus服务发现不支持数组类型的自定义标签，因此需要将每个metrics路径拆分为独立的target条目，通过labels字段指定对应的抓取路径：

[
  {"target": "1.2.3.4:10091", "labels": {"metrics_path": "/configuration/1"}},
  {"target": "1.2.3.4:10091", "labels": {"metrics_path": "/analytics/1"}},
  {"target": "1.2.3.5:10091", "labels": {"metrics_path": "/configuration/1"}},
  {"target": "1.2.3.5:10091", "labels": {"metrics_path": "/analytics/1"}},
  {"target": "1.2.3.6:10091", "labels": {"metrics_path": "/configuration/1"}},
  {"target": "1.2.3.6:10091", "labels": {"metrics_path": "/analytics/1"}}
]

2. 更新Prometheus配置 #

移除job级别的固定metrics_path，添加relabel规则将自定义的metrics_path标签映射到Prometheus内部保留标签__metrics_path__（该标签会覆盖默认的/metrics路径）：

global:
  scrape_interval: 30s

scrape_configs:
  - job_name: 'file_sd_targets'  
    file_sd_configs:
      - files:
        - /etc/prometheus/targets.json
    relabel_configs:
      - source_labels: [__address__]
        regex: '([^:]+):.*'
        target_label: instance
        replacement: '$1'
      # 将自定义标签映射为抓取路径
      - source_labels: [metrics_path]
        target_label: __metrics_path__

3. 批量生成target.json（可选） #

针对数百台服务器的场景，可通过脚本自动展开原始配置：

import json

# 原始配置，可从文件读取
original_targets = [
    {"target": "1.2.3.4:10091", "metrics_paths": ["/configuration/1", "/analytics/1"]},
    {"target": "1.2.3.5:10091", "metrics_paths": ["/configuration/1", "/analytics/1"]},
    {"target": "1.2.3.6:10091", "metrics_paths": ["/configuration/1", "/analytics/1"]}
]

expanded_targets = []
for item in original_targets:
    target_addr = item["target"]
    for path in item["metrics_paths"]:
        expanded_targets.append({
            "target": target_addr,
            "labels": {"metrics_path": path}
        })

# 写入目标文件
with open("/etc/prometheus/targets.json", "w") as f:
    json.dump(expanded_targets, f, indent=2)

内容的提问来源于stack exchange，提问作者Ankur Rangwala