如何通过Jolt Transform提取账号ID、实例ID及对应数据点

输入输出示例 #

输入JSON

{
  "1234567890": {
    "i-abc123": {
      "metrics": {
        "Datapoints": [
          {"Timestamp": "2024-01-01T00:00:00Z", "Value": 10},
          {"Timestamp": "2024-01-01T01:00:00Z", "Value": 20}
        ]
      }
    },
    "i-def456": {
      "metrics": {
        "Datapoints": []
      }
    },
    "i-ghi789": {
      "metrics": {}
    }
  }
}

预期输出JSON

[
  {
    "accountId": "1234567890",
    "instanceId": "i-abc123",
    "timestamp": "2024-01-01T00:00:00Z",
    "value": 10
  },
  {
    "accountId": "1234567890",
    "instanceId": "i-abc123",
    "timestamp": "2024-01-01T01:00:00Z",
    "value": 20
  }
]

完整Jolt规格 #

[
  // 阶段1：捕获账号ID、实例ID，提取Datapoints数组
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": {
          "metrics": {
            "Datapoints": {
              "@(3,*)": "[&3].accountId",
              "@(2,*)": "[&3].instanceId",
              "*": "[&3].datapoints[&1]"
            }
          }
        }
      }
    }
  },
  // 阶段2：标记并过滤无有效Datapoints的条目
  {
    "operation": "modify-default-beta",
    "spec": {
      "*": {
        "hasDatapoints": "=notNull(@(1,datapoints))",
        "datapointsNotEmpty": "=size(@(1,datapoints)) > 0"
      }
    }
  },
  {
    "operation": "remove",
    "spec": {
      "$[?(!(@.hasDatapoints) || !(@.datapointsNotEmpty))]": "",
      "*": {
        "hasDatapoints": "",
        "datapointsNotEmpty": ""
      }
    }
  },
  // 阶段3：将每个Datapoint拆分为独立对象，整理字段名
  {
    "operation": "shift",
    "spec": {
      "*": {
        "datapoints": {
          "*": {
            "@(2,accountId)": "[&3].accountId",
            "@(2,instanceId)": "[&3].instanceId",
            "Timestamp": "[&3].timestamp",
            "Value": "[&3].value"
          }
        }
      }
    }
  },
  // 清理空数组元素
  {
    "operation": "remove",
    "spec": {
      "$[?(@ == null)]": ""
    }
  }
]

核心实现思路 #

1. 动态键捕获：用*匹配顶级账号ID和子级实例ID，通过@(层级,*)向上回溯捕获标识信息，层级数需根据JSON结构精准计算（比如从Datapoints节点往上数3层即为顶级账号ID）。
2. 无效条目过滤：借助modify-default-beta添加辅助判断字段，再通过Jolt过滤表达式$[?(条件)]剔除Datapoints为空或不存在的实例条目。
3. 数组拆分映射：将每个实例的Datapoints数组拆分为独立对象，同时完成字段名的标准化映射（如Timestamp转timestamp）。

内容的提问来源于stack exchange，提问作者sandilya ch