Prometheus多Redis Exporter配置后Grafana仅显示部分节点问题

Let's walk through the most likely causes and fixes for your issue—since you’ve got redis_exporter instances running on ports 9121 through 9131, but only 4 are showing up in Grafana.

1. First, Confirm Prometheus Is Scraping All Targets #

Grafana can only display data that Prometheus has collected, so start here:

• Open your Prometheus web UI (typically at http://<your-prometheus-ip>:9090) and run this query:

  up{job="your-redis-exporter-job-name"}
  

  Replace your-redis-exporter-job-name with the actual job name from your /etc/prometheus/prometheus.yml config. Look for entries where up=0—these are targets Prometheus can’t reach. If your missing ports (9123-9127, 9130, 9131) aren’t even listed, double-check your scrape config:
  • Did you accidentally only include the 4 working ports in static targets?
  • If using file-based service discovery, is the target file missing entries for the other ports?
  • Are there typos (like writing 1923 instead of 9123)?

2. Verify All redis_exporter Processes Are Running #

It’s possible some exporters failed to start or crashed:

• List running redis_exporter processes:

  ps aux | grep redis_exporter
  

• Check that all expected ports are listening:

  ss -tulpn | grep redis_exporter
  

  Make sure ports 9123-9127, 9130, 9131 appear in the output.
• Test each exporter’s metrics endpoint directly:

  curl http://localhost:<port>/metrics
  

  If you get a connection error or 404, the exporter isn’t running correctly—check its startup logs for issues (like invalid flags for connecting to ElastiCache).

3. Audit Your Grafana Panel Configuration #

If Prometheus has the data but Grafana isn’t showing it, the problem is in your panel setup:

• Open the problematic Grafana panel and inspect the query editor. Look for hardcoded filters like:

  instance=~"*:1921|*:1922|*:1928|*:1929"
  

  Update the regex to include all your ports, e.g., instance=~"*:912[1-9]|*:913[0-1]".
• If using a dynamic instance variable:
  • Go to Dashboard Settings > Variables and check the variable’s query (usually something like label_values(redis_up, instance)).
  • If some instances don’t have the redis_up metric (because the exporter can’t connect to ElastiCache), they won’t show up in the variable. Fix the exporter-ElastiCache connection first.
• Double-check that the panel is using the correct Prometheus data source—accidentally selecting an old/alternate source is a common mistake.

4. Check Prometheus Scraping Logs for Errors #

Prometheus logs will tell you exactly why it’s failing to scrape a target:

• View logs (location depends on your setup—try systemd or the Prometheus log directory):

  journalctl -u prometheus.service -f
  

  Look for warnings like Error scraping target http://localhost:9123/metrics: dial tcp 127.0.0.1:9123: connect: connection refused—this will point you to the root cause (e.g., network issues, exporter crashes).

5. Ensure Network Access to AWS ElastiCache #

If the exporter is running but can’t reach ElastiCache, it won’t emit Redis metrics, which might make Grafana exclude those instances:

• Verify the exporter’s server is in a VPC/subnet that can access your ElastiCache cluster’s VPC.
• Check security groups: Make sure ElastiCache’s security group allows inbound traffic on port 6379 (default Redis port) from the exporter’s server/security group.
• Test connectivity from the exporter server:

  telnet <elasticache-endpoint> 6379
  

  If this fails, fix your VPC peering or security group rules first.

内容的提问来源于stack exchange，提问作者The Georgia