如何将Airflow指标发送至Prometheus？Grafana无数据求助

Hey there! I’ve wrestled with this exact issue before, so let me break down how to get Airflow metrics into Prometheus and fix that frustrating "no data points" error in Grafana.

First, you need to turn on Airflow’s built-in Prometheus exporter. Open your airflow.cfg file and head to the [metrics] section:

[metrics]
# Enable the prometheus endpoint
prometheus_enabled = True
# Port for the metrics endpoint (default is 9102)
metrics_port = 9102
# If you're running Airflow with multiple workers, set this directory
prometheus_multiproc_dir = /tmp/airflow_prometheus_metrics

Make sure the prometheus_multiproc_dir exists and has read/write permissions for the Airflow user. You can create it with:

mkdir -p /tmp/airflow_prometheus_metrics
chown airflow:airflow /tmp/airflow_prometheus_metrics

Restart your Airflow webserver and scheduler after making these changes. To verify the exporter is working, run:

curl http://<your-airflow-server-ip>:9102/metrics

You should see a long list of metrics starting with airflow_* if it’s working.

Next, tell Prometheus to pull metrics from Airflow. Open your prometheus.yml file and add a new scrape job under scrape_configs:

scrape_configs:
  # Existing jobs...
  - job_name: 'airflow'
    scrape_interval: 15s
    static_configs:
      - targets: ['<airflow-webserver-ip>:9102', '<airflow-scheduler-ip>:9102']

If you have Airflow workers, you can add their IPs/ports too—each worker exposes metrics on the same port. Restart Prometheus, then go to the Prometheus UI (usually at http://<prometheus-ip>:9090) and check the Status > Targets page. Your Airflow job should show as UP if the connection is working. You can also test a query like airflow_dag_run_success to confirm metrics are being scraped.

If you’re seeing this error, here are the most common fixes:

• Verify your Grafana data source: Go to Configuration > Data Sources, select your Prometheus source, and click "Test". Make sure the URL is correct and Prometheus is reachable from Grafana.
• Check your panel queries: Ensure you’re using valid Airflow metric names. For example, use airflow_task_instance_duration for task run times or airflow_dag_run_active for active DAG runs. You can find all available metrics by running the curl command from Step 1.
• Adjust the time range: If you just set this up, Grafana’s default time range (like 1 hour) might be too broad. Try selecting "Last 5 minutes" or "Last 15 minutes"—metrics won’t show up until Airflow runs some tasks.
• Trigger a test DAG: If no tasks have run since enabling metrics, there’s no data to display! Manually trigger a simple DAG (like a BashOperator that runs echo hello) and wait for it to complete. Then refresh your Grafana panel.

• Firewall/security group issues: Make sure port 9102 is open between Prometheus and Airflow servers. If you’re running in the cloud, double-check your security group rules to allow inbound traffic on 9102 from Prometheus’s IP.
• Airflow service status: If the webserver or scheduler is down, the exporter won’t run. Check their status with:

  airflow webserver status
  airflow scheduler status
  

• Prometheus target is DOWN: If the Airflow job shows as DOWN in Prometheus’s Targets page, double-check the IP/port in your prometheus.yml—typos happen! Also, confirm Airflow is actually listening on 9102 with netstat -tulpn | grep 9102.

Once you work through these steps, you should start seeing Airflow metrics populate in Grafana. If you’re still stuck, feel free to share your Airflow version, snippets of your config files, or the exact Grafana query you’re using—happy to help dig deeper!

内容的提问来源于stack exchange，提问作者Priyal Patil