Azure Docker中JMX Exporter采集Kafka指标失败，jmxterm正常

报错信息 #

SEVERE: JMX scrape failed: java.io.IOException: Failed to retrieve
RMIServer stub: javax.naming.ServiceUnavailableException [Root
exception is java.rmi.ConnectException: Connection refused to host:
localhost; nested exception is: java.net.ConnectException: Connection
refused]

环境配置 #

• Azure虚拟机：Ubuntu系统，Kafka和JMX Exporter均以Docker host网络模式运行，防火墙已开放9102、9103端口。

疑问 #

1. 为何jmxterm能正常连接并列出MBean，但JMX Exporter采集失败？
2. JMX Exporter能否绕过RMI stub直接获取Bean？
3. Azure Docker环境中运行Kafka时，如何确保RMI stub广播正确的主机名？

Kafka Broker配置 #

以Docker host模式运行，KAFKA_JMX_OPTS配置如下：

KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote \
                -Dcom.sun.management.jmxremote.port=9102 \
                -Dcom.sun.management.jmxremote.rmi.port=9102 \
                -Dcom.sun.management.jmxremote.registry.port=9102 \
                -Dcom.sun.management.jmxremote.authenticate=false \
                -Dcom.sun.management.jmxremote.ssl=false \
                -Djava.rmi.server.hostname=<Azure VM公网IP>\
                -Djava.net.preferIPv4Stack=true \
                -Dsun.rmi.transport.tcp.logLevel=VERBOSE"

JMX Exporter配置 #

docker-compose.yml #

services:
  jmx-exporter:
    network_mode: host
    container_name: jmx-exporter
    image: bitnami/jmx-exporter:latest
    volumes:
      - ./config.yml:/opt/bitnami/jmx-exporter/config.yml

config.yml #

startDelaySeconds: 0
lowercaseOutputName: false
lowercaseOutputLabelNames: false
ssl: false
jmxUrl: service:jmx:rmi:///jndi/rmi://<Azure VM公网IP>:9102/jmxrmi
hostPort: <Azure VM公网IP>:9102
rules:
  - pattern: ".*"

已尝试的排查步骤 #

• 使用jmxterm连接service:jmx:rmi:///jndi/rmi://<Azure VM公网IP>:9102/jmxrmi可正常列出所有MBean；
• 在JMX Exporter容器内用nc测试，可正常连接localhost和VM公网IP的9102端口；
• 尝试分离RMI和JMX端口（9102和9103），问题仍存在；
• 启用RMI verbose日志，显示Kafka Broker已监听预期端口。

Kafka日志片段 #

JMX启动日志 #

> broker-1  | FINE: main: localHostKnown = true, localHost =
> <Azure VM公网IP>broker-1  | FINE: RMI TCP Accept-9102: listening on port 9102

完整JMX相关启动日志 #

> broker-1  | Oct 08, 2024 4:24:21 PM sun.rmi.transport.LiveRef <init>
> broker-1  | FINE: main: localHostKnown = true, localHost = <Azure VM公网IP>
> broker-1  | Oct 08, 2024 4:24:21 PM sun.rmi.transport.tcp.TCPEndpoint getLocalEndpoint
> broker-1  | FINE: main: Version = 2, ep = [<Azure VM公网IP>:9102]
> broker-1  | Oct 08, 2024 4:24:21 PM sun.rmi.transport.tcp.TCPEndpoint getLocalEndpoint
> broker-1  | FINE: main: created local endpoint for socket factory null on port 9102
> broker-1  | Oct 08, 2024 4:24:21 PM sun.rmi.transport.tcp.TCPEndpoint getLocalEndpoint
> broker-1  | FINE: main: Version = 2, ep = [<Azure VM公网IP>:0]
> broker-1  | Oct 08, 2024 4:24:21 PM sun.rmi.transport.tcp.TCPEndpoint getLocalEndpoint
> broker-1  | FINE: main: created local endpoint for socket factory null on port 0
> broker-1  | Oct 08, 2024 4:24:21 PM sun.rmi.transport.tcp.TCPTransport exportObject
> broker-1  | FINE: main: (port 9102) create server socket
> broker-1  | Oct 08, 2024 4:24:21 PM sun.rmi.transport.tcp.TCPTransport listen
> broker-1  | FINER: main: creating server socket on [<Azure VM公网IP>:9102]
> broker-1  | Oct 08, 2024 4:24:21 PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop run
> broker-1  | FINE: RMI TCP Accept-9102: listening on port 9102
> broker-1  | Oct 08, 2024 4:24:21 PM sun.rmi.transport.tcp.TCPEndpoint getLocalEndpoint
> broker-1  | FINE: main: Version = 2, ep = [<Azure VM公网IP>:0,sun.management.jmxremote.LocalRMIServerSocketFactory@43bc63a3]
> broker-1  | Oct 08, 2024 4:24:21 PM sun.rmi.transport.LiveRef <init>
> broker-1  | FINE: main: created local endpoint for socket factory sun.management.jmxremote.LocalRMIServerSocketFactory@43bc63a3 on port 0
> broker-1  | Oct 08, 2024 4:24:21 PM sun.rmi.transport.tcp.TCPTransport exportObject
> broker-1  | FINE: main: (port 0) create server socket
> broker-1  | Oct 08, 2024 4:24:21 PM sun.rmi.transport.tcp.TCPTransport listen
> broker-1  | FINER: main: creating server socket on [<Azure VM公网IP>:0,sun.management.jmxremote.LocalRMIServerSocketFactory@43bc63a3]
> broker-1  | Oct 08, 2024 4:24:21 PM sun.rmi.transport.tcp.TCPEndpoint newServerSocket
> broker-1  | FINE: main: default port for server socket factory sun.management.jmxremote.LocalRMIServerSocketFactory@43bc63a3 and client socket factory null set to 46039
> broker-1  | Oct 08, 2024 4:24:21 PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop run
> broker-1  | FINE: RMI TCP Accept-0: listening on port 46039

核心问题 #

尽管jmxterm和连通性测试均正常，JMX Exporter仍因"Connection refused to host: localhost"报错无法采集指标。已设置java.rmi.server.hostname为VM公网IP，但RMI stub似乎仍在广播localhost。

1. 修正JMX Exporter配置中的hostPort参数 #

当前hostPort设置为VM公网IP，但在Docker host模式下，Exporter容器和Kafka共享主机网络，直接使用localhost:9102即可。修改config.yml：

# 替换原hostPort配置
hostPort: localhost:9102
# jmxUrl可改为localhost，避免公网IP路由问题
jmxUrl: service:jmx:rmi:///jndi/rmi://localhost:9102/jmxrmi

原因：同一主机的容器内，localhost访问更直接，可规避公网IP解析或网络策略带来的隐性问题。

2. 验证Kafka JMX参数的有效性 #

进入Kafka容器执行ps aux | grep kafka，检查启动参数中是否包含正确的java.rmi.server.hostname值。若参数未生效，需确认Docker启动脚本是否正确传递了该环境变量（部分Kafka镜像需通过特定方式注入JMX参数）。

3. 固定动态RMI数据端口 #

从日志可见，Kafka启动了随机动态端口（46039）作为RMI数据端口，需将其固定以避免拦截：
在KAFKA_JMX_OPTS中添加：

-Dcom.sun.management.jmxremote.rmi.port=9103

同时确保Azure防火墙开放9103端口。

4. 关于绕过RMI的问题 #

JMX Exporter目前依赖JMX RMI协议，无法直接绕过RMI stub获取MBean，必须确保RMI的主机名和端口配置完全正确才能建立连接。

内容的提问来源于stack exchange，提问作者Vasanth Nag K V