如何通过API编程式为Prometheus添加监控目标及告警？

Great question! I’ve tackled similar automation workflows where cloud orchestration tools spin up new VMs and need to automatically add them to Prometheus monitoring—here’s how to replicate (and even improve on) what you’re used to with the Zabbix API.

1. Automate Prometheus Targets: File-Based Service Discovery #

Unlike Zabbix where you call an API to add individual hosts/items, Prometheus leans into declarative configuration for targets. The simplest way to automate new VMs is using file-based service discovery:

• Have your cloud orchestration tool generate or update a JSON/YAML file (e.g., dynamic_vm_targets.json) with the new VM endpoints:

  [
    {
      "targets": ["vm-prod-01:9100", "vm-prod-02:9100", "vm-prod-03:9100"],
      "labels": {
        "job": "node_exporter",
        "env": "production",
        "orchestration_id": "deploy-12345"
      }
    }
  ]
  

• Configure Prometheus to watch this file in its prometheus.yml:

  scrape_configs:
    - job_name: 'auto_discovered_vms'
      file_sd_configs:
        - files:
          - '/etc/prometheus/dynamic_vm_targets.json'
          refresh_interval: 1m # Check for updates every minute
  

• After updating the file, trigger a Prometheus config reload to pick up changes immediately:

  curl -X POST http://your-prometheus-server:9090/-/reload
  

This works just like calling the Zabbix API to add hosts—except you’re writing to a file instead of making API calls, which is often simpler for orchestration tools like Terraform, Ansible, or custom scripts.

2. Automate Alerts & Rules #

For alerts, you can follow the same file-based pattern:

• Create or update a rules file (e.g., vm_alerts.yml) with alerts tailored to your new VMs:

  groups:
    - name: vm_high_resource_usage
      rules:
        - alert: VMHighCPU
          expr: 100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[1m])) * 100) > 80
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "High CPU usage on {{ $labels.instance }}"
            description: "{{ $labels.instance }} has CPU usage above 80% for 5 minutes."
  

• Reference this file in prometheus.yml:

  rule_files:
    - '/etc/prometheus/vm_alerts.yml'
  

• Trigger a reload to activate new alerts, just like with targets.

If you need to customize alerts per deployment, your orchestration tool can inject variables (like instance labels) into the rules file before reloading.

3. Skip Manual Files: Use Cloud-Native Service Discovery #

If you’re using a cloud provider (AWS, GCP, Azure) or Kubernetes, Prometheus has built-in service discovery integrations that eliminate the need for file updates entirely:

• For AWS EC2: Tag your new VMs with a label like monitoring: enabled, then configure Prometheus to auto-discover them:

  scrape_configs:
    - job_name: 'aws_ec2_vms'
      ec2_sd_configs:
        - region: 'us-west-2'
          access_key: '<your-access-key>'
          secret_key: '<your-secret-key>'
      relabel_configs:
        - source_labels: [__meta_ec2_tag_monitoring]
          regex: enabled
          action: keep
  

Prometheus will automatically detect new VMs with that tag and start scraping their metrics—no file writes or API calls needed.

How This Compares to Zabbix API #

Where Zabbix requires API calls to create hosts, apply templates, and add individual items, Prometheus takes a more holistic approach:

• Adding a VM to Prometheus targets automatically scrapes all metrics from its exporter (e.g., node_exporter for system metrics)—no need to define individual "monitoring items" like in Zabbix.
• Templates in Zabbix map to Prometheus scrape jobs + alert rules, which you can version-control and automate via files or service discovery.

Final Tips #

• Version-control your base Prometheus configs and rule files to track changes from orchestration deployments.
• Use tools like promtool to validate config files before reloading, to avoid breaking your Prometheus instance:

  promtool check config /etc/prometheus/prometheus.yml
  

内容的提问来源于stack exchange，提问作者Trinh Nguyen