HTTP传输下Inactivity Monitor引发异常与故障求助

问题背景 #

• 技术栈：ActiveMQ Classic 5.18.x + Camel 4.0.0，因网络限制采用HTTP Transport连接ActiveMQ
• 业务特点：流量波动极大，需维持大量连接池应对峰值，但Broker网络距离远，新建连接耗时数秒
• 故障现象：频繁出现30秒连接超时导致的写入失败；尝试通过连接字符串设置wireFormat.maxInactivityDuration和transport.useInactivityMonitor关闭Inactivity Monitor时，参数被拒绝并触发连接失败

错误日志详情 #

2024-04-19 15:13:36,717 ERROR [prm.rte.PutJMSRslt] (Camel (PrimeQueryContext) thread #261 - JmsConsumer[-queuename-])  PersonSubject Handler fails writing to -queuename for xxx/P1068EAB29ED8FQ000064289 due to: org.springframework.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: Channel was inactive (no connection attempt made) for too (>30000) long: https://xxx:443/message-queue
        at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:274)
        at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:184)
        at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:510)
        at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.send(JmsConfiguration.java:572)
        at org.apache.camel.component.jms.JmsProducer.doSend(JmsProducer.java:425)
        at org.apache.camel.component.jms.JmsProducer.processInOnly(JmsProducer.java:392)
        at org.apache.camel.component.jms.JmsProducer.process(JmsProducer.java:159)
        at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:172)
        at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:383)
        at org.apache.camel.processor.Pipeline$PipelineTask.run(Pipeline.java:102)
        at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.executeReactiveWork(DefaultReactiveExecutor.java:196)
        at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:164)
        at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:59)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:163)
        at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:383)
        at org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager.process(DefaultAsyncProcessorAwaitManager.java:81)
        at org.apache.camel.support.AsyncProcessorSupport.process(AsyncProcessorSupport.java:32)
        at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:132)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:736)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:696)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:674)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:330)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:269)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1239)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1229)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1122)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: jakarta.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: Channel was inactive (no connection attempt made) for too (>30000) long: https://xxx:443/message-queue
        at org.messaginghub.pooled.jms.pool.PooledConnection.createSession(PooledConnection.java:196)
        at org.messaginghub.pooled.jms.JmsPoolConnection.createSession(JmsPoolConnection.java:194)
        at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:213)
        at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:498)
        ... 26 more
Caused by: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: Channel was inactive (no connection attempt made) for too (>30000) long: https://xxx:443/message-queue
        at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1518)
        at org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:369)
        at org.messaginghub.pooled.jms.pool.PooledConnection.makeSession(PooledConnection.java:148)
        at org.messaginghub.pooled.jms.pool.PooledConnection$1.makeObject(PooledConnection.java:107)
        at org.messaginghub.pooled.jms.pool.PooledConnection$1.makeObject(PooledConnection.java:102)
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:780)
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:439)
        at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
        at org.messaginghub.pooled.jms.pool.PooledConnection.createSession(PooledConnection.java:178)
        ... 29 more
Caused by: org.apache.activemq.transport.InactivityIOException: Channel was inactive (no connection attempt made) for too (>30000) long: https://xxx:443/message-queue
        at org.apache.activemq.transport.AbstractInactivityMonitor$1$1.run(AbstractInactivityMonitor.java:96)
        ... 3 more

解决方案 #

1. 正确配置Inactivity Monitor参数 #

HTTP Transport不支持在连接字符串中直接传递wireFormat或Inactivity Monitor相关参数，需通过ConnectionFactory的API或Spring配置直接设置：

Java代码方式

ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory("https://xxx:443/message-queue");
// 关闭Inactivity Monitor
factory.setUseInactivityMonitor(false);
// 或调整超时时间为5分钟（根据实际网络情况设置）
factory.setMaxInactivityDuration(300000);
factory.setMaxInactivityDurationInitalDelay(60000);

Spring Boot配置方式

spring.activemq.broker-url=https://xxx:443/message-queue
# 关闭Inactivity Monitor
spring.activemq.use-inactivity-monitor=false
# 或调整超时参数
spring.activemq.max-inactivity-duration=300000
spring.activemq.max-inactivity-duration-initial-delay=60000

2. 优化连接池配置 #

针对流量波动大、连接新建耗时的场景，调整连接池参数维持活跃连接，避免闲置连接被判定为失效：

spring.activemq.pool.enabled=true
spring.activemq.pool.max-connections=100
spring.activemq.pool.min-idle=20
spring.activemq.pool.max-idle=50
# 每30秒检查一次闲置连接
spring.activemq.pool.time-between-eviction-runs=30000
# 获取连接时验证有效性
spring.activemq.pool.test-on-borrow=true
# 闲置时验证连接有效性
spring.activemq.pool.test-while-idle=true

3. 调整HTTP Transport底层超时参数 #

同步设置HTTP Transport的连接超时和读取超时，避免与Inactivity Monitor超时冲突：

连接字符串方式

https://xxx:443/message-queue?transport.soTimeout=60000&transport.connectTimeout=10000

Java代码方式

ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory();
factory.setBrokerURL("https://xxx:443/message-queue");
Map<String, String> transportParams = new HashMap<>();
transportParams.put("soTimeout", "60000"); // 读取超时60秒
transportParams.put("connectTimeout", "10000"); // 连接超时10秒
factory.setTransportParameters(transportParams);

4. 启用自动重连机制 #

配置自动重连，当连接因超时失效时自动重建，减少业务写入失败：

Spring Boot配置方式

spring.activemq.reconnection-delay=1000
spring.activemq.max-reconnection-delay=30000
spring.activemq.reconnect-attempts=-1 # 无限次重连

Java代码方式

factory.setReconnectionDelay(1000);
factory.setMaxReconnectionDelay(30000);
factory.setReconnectAttempts(-1);

内容的提问来源于stack exchange，提问作者Clint