JMX Agent服务端程序内部配置SSL属性启动报SSL握手错误求助

问题根因 #

你遇到的Unsupported or unrecognized SSL message错误核心原因是手动创建的RMI Registry默认使用明文Socket，没有绑定SSL配置，仅给JMX Connector Server配置SSL Server Socket Factory不会生效：客户端发送SSL请求到RMI Registry时，Registry返回明文响应，SSL层无法识别报文格式就会抛出该错误。

必要调整步骤 #

1. 修正RMI Registry创建逻辑 #

替换默认的明文Registry创建逻辑，手动指定和JMX服务匹配的SSL Socket工厂，开启双向认证：

// 构造SSL Socket工厂，复用系统属性配置的keystore/truststore，开启客户端认证
SslRMIServerSocketFactory serverSockFactory = new SslRMIServerSocketFactory(
    null,
    null,
    true // 对应原配置的com.sun.management.jmxremote.ssl.need.client.auth=true
);
SslRMIClientSocketFactory clientSockFactory = new SslRMIClientSocketFactory();

// 创建带SSL配置的RMI Registry
LocateRegistry.createRegistry(
    port,
    clientSockFactory,
    serverSockFactory
);

2. 同步JMX Connector环境配置 #

给JMX Connector的环境变量补充完整的Socket工厂配置，避免依赖全局系统属性的优先级问题：

env.put(RMIConnectorServer.RMI_SERVER_SOCKET_FACTORY_ATTRIBUTE, serverSockFactory);
env.put(RMIConnectorServer.RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE, clientSockFactory);
// 直接在环境变量中配置认证规则，优先级高于全局系统属性
env.put("com.sun.management.jmxremote.ssl", "true");
env.put("com.sun.management.jmxremote.ssl.need.client.auth", "true");
env.put("com.sun.management.jmxremote.authenticate", "false");

3. 调整系统属性设置时机 #

所有System.setProperty配置keystore、truststore、TLS协议的代码，必须放在SSL Socket工厂实例化之前执行，否则SSL上下文初始化时会读取不到配置。

4. 清理无效配置 #

你代码中重复定义了两次boolean pwd_auth = false;，删除多余的定义即可；手动创建JMX Connector的场景下，com.sun.management.jmxremote.port、com.sun.management.jmxremote.rmi.port两个属性不需要设置，无实际作用。

完整可运行的服务端代码示例 #

boolean pwd_auth = false;
boolean ssl_auth = true;
String strPort = "9999";
int port = 9999;
Map<String,Object> env = new HashMap<>();

// 第一步：先设置所有SSL相关系统属性，必须在SocketFactory实例化前执行
System.setProperty("javax.net.ssl.keyStore", Config.keyStoreFile);
System.setProperty("javax.net.ssl.keyStorePassword", Config.keyStorePwd );
System.setProperty("javax.net.ssl.trustStore" , Config.trustStoreFile );
System.setProperty("javax.net.ssl.trustStorePassword" , Config.trustStorePwd);
System.setProperty("jdk.tls.server.protocols", Config.protocol);

// 第二步：构造SSL Socket工厂
SslRMIServerSocketFactory serverSockFactory = new SslRMIServerSocketFactory(
    null,
    null,
    ssl_auth
);
SslRMIClientSocketFactory clientSockFactory = new SslRMIClientSocketFactory();

// 第三步：创建带SSL的RMI Registry
LocateRegistry.createRegistry(
    port,
    clientSockFactory,
    serverSockFactory
);

// 第四步：配置JMX Connector环境参数
env.put(RMIConnectorServer.RMI_SERVER_SOCKET_FACTORY_ATTRIBUTE, serverSockFactory);
env.put(RMIConnectorServer.RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE, clientSockFactory);
env.put("com.sun.management.jmxremote.ssl", String.valueOf(ssl_auth));
env.put("com.sun.management.jmxremote.ssl.need.client.auth", String.valueOf(ssl_auth));
env.put("com.sun.management.jmxremote.authenticate", String.valueOf(pwd_auth));

// 第五步：启动JMX Connector Server
MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
String strUrl = "service:jmx:rmi:///jndi/rmi://:"+port+"/jmxrmi";
JMXServiceURL url = new JMXServiceURL(strUrl);
JMXConnectorServer cs = JMXConnectorServerFactory.newJMXConnectorServer(url, env, mbs);
cs.start();

// 注册MBean
Hello mbean = new Hello();
ObjectName name = new ObjectName(Config.oname);
mbs.registerMBean(mbean, name);

内容的提问来源于stack exchange，提问作者shibendra chakraborty