如何为Prometheus Consul目标创建非全健康状态的Grafana告警？

Absolutely! You can absolutely set up this alert—let me walk you through the exact steps, since we can calculate the healthy target count using existing Prometheus metrics instead of needing a dedicated "healthy targets" gauge.

Core Idea: Use the up Metric to Track Healthy Targets #

Prometheus automatically exposes the up metric for every target it scrapes:

• A value of 1 means the target is healthy (Prometheus successfully completed a scrape)
• A value of 0 means the target is unhealthy (scrape failed for any reason)

We can pair this with your existing prometheus_sd_discovered_targets metric to compare healthy vs. total targets.

Step 1: Validate Your Metrics #

First, confirm the metrics behave as expected in the Prometheus expression browser:

• Run up{job="your-consul-job"} (replace your-consul-job with your actual job name). You’ll see a time series for each Consul target, with 1 or 0 indicating health.
• Run prometheus_sd_discovered_targets{job="your-consul-job"}—this number should match the total target count shown on Prometheus’ Targets page.

Step 2: Write the Alert PromQL Expression #

The goal is to trigger an alert when the count of healthy targets doesn’t equal the total discovered targets. Here’s the base expression:

sum(up{job="your-consul-job"}) != prometheus_sd_discovered_targets{job="your-consul-job"}

To avoid false alerts when there are zero targets configured, add a guard clause:

sum(up{job="your-consul-job"}) != prometheus_sd_discovered_targets{job="your-consul-job"} AND prometheus_sd_discovered_targets{job="your-consul-job"} > 0

For Granular, Service-Specific Alerts #

If your Consul targets are grouped by a label like service, you can split the alert to pinpoint exactly which service has issues:

sum by (service) (up{job="your-consul-job"}) != sum by (service) (prometheus_sd_discovered_targets{job="your-consul-job"}) AND sum by (service) (prometheus_sd_discovered_targets{job="your-consul-job"}) > 0

Step 3: Configure the Alert in Grafana #

1. Navigate to Grafana’s Alerting section and create a new Alert Rule.
2. Select your Prometheus data source, then paste the PromQL expression you built.
3. Set the alert condition: For example, trigger the alert when the expression evaluates to true for at least 1 minute (this prevents flapping from temporary network blips).
4. Customize the alert notification with clear context, like:

   Alert: Unhealthy Consul Targets Detected
   Job {{ $labels.job }} has {{ prometheus_sd_discovered_targets{job="your-consul-job"} }} total targets, but only {{ sum(up{job="your-consul-job"}) }} are currently healthy.

5. Attach your preferred notification channels (Slack, email, etc.) to ensure you get alerted promptly.

Pro Tips #

• Build a quick Grafana panel to visualize sum(up{job="your-consul-job"}) and prometheus_sd_discovered_targets{job="your-consul-job"} side by side first. This helps confirm the metrics align correctly before setting up alerts.
• Double-check the job label filter to make sure you’re only targeting your Consul-discovered targets (avoid accidentally including other scrape jobs).

内容的提问来源于stack exchange，提问作者Gijs de Jong