使用JGroups时ActiveMQ Artemis消息Scale Down失败

问题描述 #

使用JGroups搭建ActiveMQ Artemis（v2.29.0）集群，通过发现组启用Scale Down高可用策略。执行artemis stop命令停止broker时，Scale Down功能失败，但使用连接器测试时功能正常。

broker.xml配置 #

<acceptors>
   <acceptor name="artemis">tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true;supportAdvisory=false;suppressInternalManagementObjects=false</acceptor>
</acceptors>

<connectors>
   <connector name="netty-connector">tcp://0.0.0.0:61616</connector>
</connectors>

<broadcast-groups>
   <broadcast-group name="my-broadcast-group">
      <broadcast-period>5000</broadcast-period>
      <jgroups-file>jgroup-file-ping.xml</jgroups-file>
      <jgroups-channel>active_broadcast_channel</jgroups-channel>
      <connector-ref>netty-connector</connector-ref>
   </broadcast-group>
</broadcast-groups>

<discovery-groups>
   <discovery-group name="my-discovery-group">
      <jgroups-file>jgroup-file-ping.xml</jgroups-file>
      <jgroups-channel>active_broadcast_channel</jgroups-channel>
      <refresh-timeout>10000</refresh-timeout>
   </discovery-group>
</discovery-groups>

<cluster-connections>
   <cluster-connection name="my-cluster">
      <connector-ref>netty-connector</connector-ref>
      <retry-interval>2000</retry-interval>
      <use-duplicate-detection>true</use-duplicate-detection>
      <message-load-balancing>ON_DEMAND</message-load-balancing>
      <max-hops>1</max-hops>
      <discovery-group-ref discovery-group-name="my-discovery-group"/>
   </cluster-connection>
</cluster-connections>

<ha-policy>
   <live-only>
      <scale-down>
         <enabled>true</enabled>
         <discovery-group-ref discovery-group-name="my-discovery-group"/>
      </scale-down>
   </live-only>
</ha-policy>

jgroup-file-ping.xml配置 #

<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="urn:org:jgroups"
        xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
    <TCP bind_addr="${jgroups.bind_addr:site_local}"
         bind_port="${jgroups.bind_port:7800}"
         external_addr="${jgroups.external_addr}"
         external_port="${jgroups.external_port}"
         thread_pool.min_threads="0"
         thread_pool.max_threads="200"
         thread_pool.keep_alive_time="30000"/>
    <RED/>

    <!-- 所有broker均可访问的共享目录 -->
    <FILE_PING location="../../file.ping.dir"/>
    <MERGE3  min_interval="10000"
             max_interval="30000"/>
    <FD_SOCK2/>
    <FD_ALL3 timeout="40000" interval="5000" />
    <VERIFY_SUSPECT2 timeout="1500"  />
    <BARRIER />
    <pbcast.NAKACK2 use_mcast_xmit="false" />
    <UNICAST3 />
    <pbcast.STABLE desired_avg_gossip="50000"
                   max_bytes="4M"/>
    <pbcast.GMS print_local_addr="true" join_timeout="2000"/>
    <UFC max_credits="2M"
         min_threshold="0.4"/>
    <MFC max_credits="2M"
         min_threshold="0.4"/>
    <FRAG2 frag_size="60K"  />
    <!--RSVP resend_interval="2000" timeout="10000"/-->
    <pbcast.STATE_TRANSFER/>
</config>

报错日志 #

2023-07-31 17:01:15,202 INFO  [org.apache.activemq.artemis.core.server] AMQ221029: stopped bridge $.artemis.internal.sf.my-cluster.eef60017-2f81-11ee-adb4-ea6113f38d3a
2023-07-31 17:01:15,295 WARN  [org.apache.activemq.artemis.core.server] AMQ222181: Unable to scaleDown messages
org.apache.activemq.artemis.api.core.ActiveMQInternalErrorException: AMQ219004: Failed to initialise session factory
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.initialize(ServerLocatorImpl.java:277) ~[artemis-core-client-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:671) ~[artemis-core-client-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:570) ~[artemis-core-client-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549) ~[artemis-core-client-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.core.server.LiveNodeLocator.connectToCluster(LiveNodeLocator.java:85) ~[artemis-server-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.connectToScaleDownTarget(LiveOnlyActivation.java:148) [artemis-server-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.freezeConnections(LiveOnlyActivation.java:116) [artemis-server-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.freezeConnections(ActiveMQServerImpl.java:1550) [artemis-server-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1324) [artemis-server-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1243) [artemis-server-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1227) [artemis-server-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1038) [artemis-server-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.integration.FileBroker.stop(FileBroker.java:94) [artemis-cli-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.cli.commands.Run.stop(Run.java:222) [artemis-cli-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.cli.commands.Run$2.run(Run.java:196) [artemis-cli-2.29.0.jar:2.29.0]
        at java.util.TimerThread.mainLoop(Timer.java:566) [?:?]
        at java.util.TimerThread.run(Timer.java:516) [?:?]
Caused by: org.apache.activemq.artemis.api.core.ActiveMQInternalErrorException: channel is closed
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.startDiscovery(ServerLocatorImpl.java:291) ~[artemis-core-client-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.initialize(ServerLocatorImpl.java:273) ~[artemis-core-client-2.29.0.jar:2.29.0]
        ... 16 more
Caused by: java.lang.IllegalStateException: channel is closed
        at org.jgroups.JChannel.checkClosed(JChannel.java:983) ~[jgroups-5.2.0.Final.jar:5.2.0.Final]
        at org.jgroups.JChannel._preConnect(JChannel.java:792) ~[jgroups-5.2.0.Final.jar:5.2.0.Final]
        at org.jgroups.JChannel.connect(JChannel.java:322) ~[jgroups-5.2.0.Final.jar:5.2.0.Final]
        at org.jgroups.JChannel.connect(JChannel.java:316) ~[jgroups-5.2.0.Final.jar:5.2.0.Final]
        at org.apache.activemq.artemis.api.core.jgroups.JChannelWrapper.connect(JChannelWrapper.java:127) ~[artemis-core-client-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.api.core.JGroupsBroadcastEndpoint.internalOpen(JGroupsBroadcastEndpoint.java:121) ~[artemis-core-client-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.api.core.JGroupsBroadcastEndpoint.openClient(JGroupsBroadcastEndpoint.java:99) ~[artemis-core-client-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.core.cluster.DiscoveryGroup.start(DiscoveryGroup.java:113) ~[artemis-core-client-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.startDiscovery(ServerLocatorImpl.java:289) ~[artemis-core-client-2.29.0.jar:2.29.0]
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.initialize(ServerLocatorImpl.java:273) ~[artemis-core-client-2.29.0.jar:2.29.0]
        ... 16 more
2023-07-31 17:01:15,485 INFO  [org.apache.activemq.artemis] AMQ241005: Stopping embedded web server
2023-07-31 17:01:15,497 INFO  [io.hawt.web.auth.AuthenticationFilter] Destroying hawtio authentication filter
2023-07-31 17:01:15,499 INFO  [io.hawt.HawtioContextListener] Destroying hawtio services
2023-07-31 17:01:15,530 INFO  [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed artemis-plugin plugin
2023-07-31 17:01:15,542 INFO  [org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed activemq-branding plugin
2023-07-31 17:01:15,568 INFO  [org.apache.activemq.artemis] AMQ241006: Stopped embedded web server
2023-07-31 17:01:15,570 INFO  [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.29.0 [2df21bb4-24c6-11ee-accf-0adf5b5db694] stopped, uptime 13.875 seconds

解决方法 #

问题根源 #

broker停止流程中，JGroups通道会被提前关闭，导致Scale Down执行时无法通过发现组查找集群内的目标节点，最终初始化会话工厂失败。

方案1：改用静态连接器配置Scale Down（推荐） #

放弃发现组，直接在Scale Down配置中指定目标broker的静态连接器，避免依赖JGroups通道。

修改broker.xml的ha-policy部分：

<ha-policy>
   <live-only>
      <scale-down>
         <enabled>true</enabled>
         <connectors>
            <connector-ref>target-broker-connector</connector-ref>
         </connectors>
      </scale-down>
   </live-only>
</ha-policy>

同时在connectors节点下添加目标broker的连接器：

<connectors>
   <connector name="netty-connector">tcp://0.0.0.0:61616</connector>
   <!-- 替换为目标broker的实际地址和端口 -->
   <connector name="target-broker-connector">tcp://192.168.1.100:61617</connector>
</connectors>

方案2：调整broker停止顺序（临时 workaround） #

修改Artemis的停止逻辑，确保Scale Down操作完成后再关闭JGroups通道。此方法需要自定义或修改Artemis源码，复杂度较高，仅作为备选方案。

额外检查项 #

• 确认jgroup-file-ping.xml中FILE_PING指定的目录，所有集群broker都拥有读写权限，保证集群发现正常。
• 确保两台broker的JGroups通道名称（active_broadcast_channel）一致，避免集群分裂。

内容的提问来源于stack exchange，提问作者Subhidh Agarwal