使用JMX与CloudWatch Agent传输Kafka/Zookeeper指标至CloudWatch遇配置验证问题

问题场景 #

已为Kafka和Zookeeper配置JMX，尝试通过CloudWatch Agent将指标发送至AWS CloudWatch，但执行配置加载命令时出现JSON Schema验证错误。

现有配置 #

Kafka JMX配置

export KAFKA_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Dcom.sun.management.jmxremote.host=localhost -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" 

Zookeeper使用默认JMX端口9998。

错误的CloudWatch Agent配置文件

{
  "agent": {
    "metrics_collection_interval": 60,
    "region": "us-east-1"
  },
  "metrics": {
    "namespace": "KZMetrics",
    "metrics_collected": {
      "jmx": {
        "kafka": {
          "jmx_endpoint": "localhost:9999",
          "measurement": [
            {
              "name": "java.lang:type=OperatingSystem/SystemCpuLoad",
              "alias": "Kafka_CPUUsage",
              "unit": "Percent"
            },
            {
              "name": "java.lang:type=Memory/HeapMemoryUsage",
              "alias": "Kafka_MemoryUsage",
              "unit": "Bytes"
            }
          ],
          "metrics_collection_interval": 60
        }
      }
    }
  }
}

启动命令

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

错误信息

2024/08/25 08:10:45 E! Invalid Json input schema.
2024/08/25 08:10:45 E! Invalid Json input schema.
2024/08/25 08:10:45 Under path : /metrics/metrics_collected/jmx | Error : Must validate one and only one schema (oneOf)
2024/08/25 08:10:45 Under path : /metrics/metrics_collected/jmx | Error : Must validate at least one schema (anyOf)
2024/08/25 08:10:45 Under path : /metrics/metrics_collected/jmx | Error : jvm is required
2024/08/25 08:10:45 Under path : /metrics/metrics_collected/jmx | Error : endpoint is required
2024/08/25 08:10:45 Under path : /metrics/metrics_collected/jmx | Error : Additional property jmx_endpoint is not allowed

Agent状态

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status

输出：

{
  "status": "running",
  "starttime": "2024-08-25T07:42:10+00:00",
  "configstatus": "configured",
  "version": "1.300043.0b781"
}

错误原因分析 #

从错误日志可明确几个核心问题：

1. JMX配置结构错误：不应在jmx节点下嵌套自定义的kafka子节点，需直接在jmx层级配置JMX端点参数
2. 字段名错误：使用了不被允许的jmx_endpoint字段，正确字段为endpoint
3. 缺少必填字段：未指定jvm字段（用于标识JVM实例，方便区分不同服务的指标）

修正后的配置 #

仅收集Kafka指标的配置 #

{
  "agent": {
    "metrics_collection_interval": 60,
    "region": "us-east-1"
  },
  "metrics": {
    "namespace": "KZMetrics",
    "metrics_collected": {
      "jmx": {
        "endpoint": "localhost:9999",
        "jvm": "kafka",
        "measurement": [
          {
            "name": "java.lang:type=OperatingSystem/SystemCpuLoad",
            "alias": "Kafka_CPUUsage",
            "unit": "Percent"
          },
          {
            "name": "java.lang:type=Memory/HeapMemoryUsage",
            "alias": "Kafka_MemoryUsage",
            "unit": "Bytes"
          }
        ],
        "metrics_collection_interval": 60
      }
    }
  }
}

同时收集Kafka与Zookeeper指标的配置 #

若需同时收集两个服务的JMX指标，需将jmx配置为数组，每个元素对应一个服务的JMX端点：

{
  "agent": {
    "metrics_collection_interval": 60,
    "region": "us-east-1"
  },
  "metrics": {
    "namespace": "KZMetrics",
    "metrics_collected": {
      "jmx": [
        {
          "endpoint": "localhost:9999",
          "jvm": "kafka",
          "measurement": [
            {
              "name": "java.lang:type=OperatingSystem/SystemCpuLoad",
              "alias": "Kafka_CPUUsage",
              "unit": "Percent"
            },
            {
              "name": "java.lang:type=Memory/HeapMemoryUsage",
              "alias": "Kafka_MemoryUsage",
              "unit": "Bytes"
            }
          ],
          "metrics_collection_interval": 60
        },
        {
          "endpoint": "localhost:9998",
          "jvm": "zookeeper",
          "measurement": [
            {
              "name": "java.lang:type=OperatingSystem/SystemCpuLoad",
              "alias": "ZooKeeper_CPUUsage",
              "unit": "Percent"
            },
            {
              "name": "java.lang:type=Memory/HeapMemoryUsage",
              "alias": "ZooKeeper_MemoryUsage",
              "unit": "Bytes"
            }
          ],
          "metrics_collection_interval": 60
        }
      ]
    }
  }
}

验证步骤 #

1. 将修正后的配置文件替换到/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
2. 重新加载配置并重启Agent：

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

3. 检查Agent状态，确认configstatus为configured且无报错：

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status

4. 登录AWS CloudWatch控制台，在指标命名空间KZMetrics下查看上报的指标。

内容的提问来源于stack exchange，提问作者Nagaraj