Windows环境Hadoop配置JMX Exporter对接Prometheus遇错求助

Hey there! Let's break down your JMX Exporter issue first, then cover alternative monitoring options for your Windows Hadoop POC.

1. Fixing the JMX Exporter Configuration Error #

That NumberFormatException is the key here—your javaagent parameter formatting is tripping up Windows cmd. Here's why and how to fix it:

• The error happens because the extra quotes around your javaagent argument are causing the JVM to interpret "C" as the port number (instead of 1985). Windows cmd handles environment variables and quotes differently than Unix shells.
• Update your hadoop_installation\bin\hdfs.cmd line to this (remove the outer quotes around the entire javaagent string):

  set HADOOP_OPTS=%HADOOP_OPTS% -javaagent:C:\path\to\jmx_prometheus_javaagent-0.11.0.jar=1985:C:\path\to\conf\hadoop.yml
  

  If your paths have spaces, wrap individual paths in quotes instead:

  set HADOOP_OPTS=%HADOOP_OPTS% -javaagent:"C:\My Folder\jmx_prometheus_javaagent-0.11.0.jar"=1985:"C:\My Folder\conf\hadoop.yml"
  

• Also, note that start-all.cmd is deprecated—for testing, use start-dfs.cmd and start-yarn.cmd separately to isolate service startup issues.

Fix Your hadoop.yml Rule Too #

Your YAML pattern has unescaped special characters that might cause parsing issues. Wrap the pattern in single quotes and use a standard Prometheus metric name:

- pattern: 'Hadoop<service=ResourceManager, name=MetricsSystem, sub=Stats><>NumAllSources'
  name: hadoop_rm_total_sources
  labels:
    app_id: "hadoop_rm"

2. Alternative Monitoring Methods (Beyond FSImage Exporter) #

If you want other options for monitoring Windows Hadoop:

• Hadoop Metrics2 + Prometheus Sink: Hadoop's built-in Metrics2 framework can push metrics directly to Prometheus. Edit hadoop-metrics2.properties to add a Prometheus sink:

  *.sink.prometheus.class=org.apache.hadoop.metrics2.sink.prometheus.PrometheusMetricsSink
  *.sink.prometheus.port=9091
  

  Ensure your Hadoop classpath includes the Prometheus sink JAR (most modern Hadoop distributions include this by default).
• Grafana Agent JMX Scraping: Grafana Agent has native JMX support and a Windows installer. You can configure it to scrape Hadoop's JMX endpoints directly and forward metrics to Prometheus—no need to mess with javaagents on every Hadoop service.
• PowerShell + Pushgateway: Write simple PowerShell scripts to run Hadoop CLI commands (like hdfs dfsadmin -report or yarn node -list), parse the output into Prometheus-compatible metrics, and push them to a Pushgateway. This is great for collecting operational metrics that JMX doesn't cover.

3. Quick Validation Steps #

After fixing the config:

1. Start your Hadoop services with start-dfs.cmd and start-yarn.cmd
2. Visit http://localhost:1985/metrics (or your chosen port) to confirm JMX Exporter is serving metrics
3. Check Hadoop's log files (in hadoop_installation\logs) for any remaining errors if things still don't work

内容的提问来源于stack exchange，提问作者Rahul Borkar