容器化Prometheus无法抓取JMX-exporter暴露的Spark应用指标问题求助

Let’s break down the possible issues here since you’ve confirmed the JMX exporter is correctly exposing metrics via http://localhost:8888/metrics — that’s a solid first step! Below are the most common reasons Prometheus can’t reach your Spark target, along with actionable fixes:

1. Test Basic Network Connectivity from Prometheus Server #

First, rule out network-level issues by directly accessing the metrics endpoint from the machine where Prometheus runs:

curl http://<spark-driver-ip>:8888/metrics

If this command fails (timeout, connection refused):

• If Spark and Prometheus are on different machines: localhost in your Prometheus config won’t work — replace it with the actual IP address of the machine running Spark.
• Check firewalls/security groups: Ensure port 8888 is open between the two machines. On Linux, you can temporarily allow traffic for testing with:

  sudo iptables -A INPUT -p tcp --dport 8888 -j ACCEPT
  

2. Fix JMX Exporter’s Listening Address #

By default, the JMX Prometheus Java Agent binds only to 127.0.0.1, which blocks external connections (even from other processes on the same machine using a different network interface). To allow connections from any source, specify 0.0.0.0 as the bind address when starting Spark:

spark-shell --conf "spark.driver.extraJavaOptions=-javaagent:jmx_prometheus_javaagent-0.10.jar=0.0.0.0:8888:.../spark.yml"

Verify the change with this command — you should see the agent listening on all interfaces:

ss -tulpn | grep 8888

A valid output looks like:

LISTEN 0      100    0.0.0.0:8888      0.0.0.0:*    users:(("java",pid=1234,fd=15))

3. Confirm Spark Driver’s Actual IP Address #

Spark sometimes binds to a non-localhost IP (e.g., if your machine has multiple network interfaces). Check the Spark driver logs for a line like:

Started SparkUI at http://192.168.1.100:4040

Use that same IP address in your Prometheus targets configuration instead of localhost.

4. Check Prometheus Logs for Specific Errors #

Dig into Prometheus logs (usually located in /var/log/prometheus/ or the directory where you launched Prometheus) for clues about the failed scrape. Common errors include:

• context deadline exceeded: Prometheus can’t establish a connection to the endpoint (network timeout).
• connection refused: Nothing is listening on port 8888, or the port is blocked.
  These logs will pinpoint exactly what’s going wrong.

5. Validate and Reload Prometheus Configuration #

Double-check your prometheus.yml for typos (e.g., missing commas, incorrect port numbers). After making changes, restart Prometheus to apply updates:

# If using systemd
sudo systemctl restart prometheus

# If running Prometheus directly
kill -HUP <prometheus-pid>

Once you resolve the underlying issue, refresh the Prometheus targets page — your spark_streaming_app should show as UP.

内容的提问来源于stack exchange，提问作者Oded