JMX Exporter监控ActiveMQ Artemis集群节点失联问题求助

我有一个3节点的ActiveMQ Artemis集群，偶尔会出现单个节点无法识别其他节点但仍能独立运行的情况，导致集群分裂为正常集群和孤立节点两部分。

我想通过每个节点上部署的JMX Exporter实现该场景的监控告警，目前仅发现JMX中的Topology属性可查看集群成员，但JMX Exporter无法抓取这个属性。我知道监控指标需要数值类型，所以测试规则里把value设为1，规则配置如下：

- pattern: "^org.apache.activemq.artemis<broker=\"([^\"]*)\",\\s*component=cluster-connections,\\s*name=\"([^\"]*)\"><>([^:]*):(.*)"
    attrNameSnakeCase: true
    name: artcluster_$3
    value: 1
    labels:
        a1: $1
        a2: $2
        a3: $3
        a4: $4  

相关属性情况如下（标注了可抓取和不可抓取的属性）：

scrape            "Started": true,
scrape            "Address": "",
scrape            "MessagesAcknowledged": 1,
scrape            "MessageLoadBalancingType": "ON_DEMAND",
!DONT scrape      "Topology": "topology on Topology@47f077e3[owner=ClusterConnectionImpl@53887553[nodeUUID=e6b1d33c-a8af-11ed-837e-0242ac110002, connector=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=art-smx03-t, address=, server=ActiveMQServerImpl::name=art-smx03-t]]:\n\te6c92b4c-a8af-11ed-baf9-0242ac110002 => TopologyMember[id=e6c92b4c-a8af-11ed-baf9-0242ac110002, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=app-smx01-t, b=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=62616&host=art-smx03-t], backupGroupName=art-smx03-t, scaleDownGroupName=null]\n\te5777e6d-a8af-11ed-887a-0242ac110002 => TopologyMember[id=e5777e6d-a8af-11ed-887a-0242ac110002, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=art-smx01-t, b=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=62616&host=app-smx01-t-], backupGroupName=app-smx01-t, scaleDownGroupName=null]\n\te6b1d33c-a8af-11ed-837e-0242ac110002 => TopologyMember[id=e6b1d33c-a8af-11ed-837e-0242ac110002, connector=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=art-smx03-t, b=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=62616&host=art-smx01-t], backupGroupName=art-smx01-t, scaleDownGroupName=null]\n\tnodes=6\tmembers=3",
scrape            "MaxHops": 1,
!DONT scrape      "Nodes": {
                                  "e5777e6d-a8af-11ed-887a-0242ac110002": "art-smx01-t",
                                  "e6c92b4c-a8af-11ed-baf9-0242ac110002": "app-smx01-t"
                                         },
scrape            "Name": "smx-art-cluster",
scrape            "DuplicateDetection": true,

所有属性层级相同，数值、布尔类型及短文本属性能正常抓取，但长文本属性不行。Topology是java.lang.String类型却无法抓取，而同为java.lang.String的短文本属性MessageLoadBalancingType可以正常抓取。当前使用JMX Exporter版本是0.18.0，请问怎么解决这个问题来实现集群节点失联的监控告警？

1. 调整JMX Exporter配置，提取Topology中的关键数值 #

JMX Exporter 0.18.0对长文本属性的支持有限，但可以通过自定义规则匹配Topology属性，并从中提取成员数量这类关键数值：

- pattern: "^org.apache.activemq.artemis<broker=\"([^\"]*)\",\\s*component=cluster-connections,\\s*name=\"([^\"]*)\"\\><>Topology:(.*)"
    name: artemis_cluster_member_count
    value: "$3"
    labels:
      broker: "$1"
      cluster_connection: "$2"
    type: GAUGE
    transform:
      script: |
        def matcher = java.util.regex.Pattern.compile('members=(\\d+)').matcher(value);
        if (matcher.find()) {
            return Integer.parseInt(matcher.group(1));
        }
        return 0;

注意：此配置需要JMX Exporter启用Groovy脚本支持，若0.18.0版本未包含相关依赖，可升级至0.20+版本，新版本对长文本属性处理更友好，且脚本transform功能更稳定。

2. 改用Nodes属性统计集群节点数量 #

虽然Nodes是Map类型，但可以通过调整规则将其拆分为多个指标实例，通过统计指标数量判断集群状态：

- pattern: "^org.apache.activemq.artemis<broker=\"([^\"]*)\",\\s*component=cluster-connections,\\s*name=\"([^\"]*)\"\\><>Nodes\\[(.*)\\]:(.*)"
    name: artemis_cluster_node_present
    value: 1
    labels:
      broker: "$1"
      cluster_connection: "$2"
      node_uuid: "$3"
      node_name: "$4"
    type: GAUGE

正常3节点集群中，每个节点应抓取到2个其他节点的指标；当节点失联时，该节点抓取到的指标数量会减少，据此可配置告警。

3. 升级JMX Exporter版本 #

0.18.0版本较旧，后续版本（如0.20.0+）修复了长文本属性抓取的相关问题，升级后可能无需额外配置即可正常抓取Topology属性。

4. 配置Prometheus告警规则 #

无论采用哪种方式获取集群成员数据，都需要配置告警规则触发通知。示例规则如下：

groups:
- name: artemis_cluster_alerts
  rules:
  - alert: ArtemisClusterNodeIsolation
    expr: artemis_cluster_member_count < 2
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Artemis节点 {{ $labels.broker }} 集群成员数量异常"
      description: "节点 {{ $labels.broker }} 当前仅识别到 {{ $value }} 个集群成员，预期为2个，可能已被孤立。"

内容的提问来源于stack exchange，提问作者Mihail Ma