如何修改Prometheus已采集指标的Job名称以保留Grafana历史图表？

Great question—this is a super common pain point when renaming scrape jobs in Prometheus, since its TSDB is built around immutable time series (you can’t directly edit existing metrics stored on disk). Here are your practical options to fix the Grafana historical chart gap:

1. Quick Fix: Update Grafana Queries (No Data Modification Needed) #

If your main goal is just to get Grafana showing both historical and new data, this is the fastest route. Instead of modifying metrics, adjust your Grafana panel queries to match both the old and new job names using a regex match:

• Change queries like job="old-scrape-job" to job=~"old-scrape-job|new-scrape-job"
• This will pull in both historical (old job) and current (new job) data. Once you stop the old scrape task, you can update the queries to only target the new job name.

2. Rewrite Historical Data with Remote Write + Relabeling #

If you need to permanently standardize the job label across all historical data, use Prometheus’s remote write functionality with a relabeling proxy:

• Step 1: Deploy a simple relabeling tool (like a custom lightweight service or open-source tools built for this purpose) that intercepts remote write requests.
• Step 2: Configure your existing Prometheus instance to send data to this proxy, with a rule to rewrite the job label from the old name to the new name.
• Step 3: Configure the proxy to send the modified data back to your Prometheus instance (or a new one). Be careful to avoid duplicate time series—you may want to temporarily stop the new scrape job while processing historical data, then resume it once done.

3. Use Thanos Rewrite (For Long-Term Object Storage) #

If you’re using Thanos with object storage (like S3, GCS) for long-term metric retention, the thanos rewrite command is purpose-built for this scenario:

• Run a command like this to batch-update the job label across all historical metrics:

  thanos rewrite \
    --objstore.config-file=your-object-store-config.yaml \
    --rule 'replace job="old-job-name" "new-job-name"'
  

• This modifies the labels directly in your object storage, so Thanos Query will return unified results with the new job name for both historical and current data.

Key Notes: #

• Since you’ve been running both jobs for a month, make sure to handle overlapping time series (if any) to avoid duplicate data points in your final dataset.
• Prometheus itself doesn’t support in-place label edits, so all solutions involve exporting, modifying, and re-ingesting data (or using a layered tool like Thanos).

内容的提问来源于stack exchange，提问作者Mike Zhang