如何使用NiFi Jolt将嵌套数组转换为自定义对象数组

需求说明 #

需要将输入JSON中states字段的嵌套数组，转换为包含指定键名的对象数组，键名需手动指定，具体列名如下：

{
  "column_names": [
    "icao24",
    "callsign",
    "origin_country",
    "time_position",
    "last_contact",
    "long",
    "lat",
    "baro_altitude",
    "on_ground",
    "velocity",
    "true_track",
    "vertical_rate",
    "sensors",
    "geo_altitude",
    "squawk",
    "spi",
    "position_source"
  ]
}

示例输入数据 #

{
  "time": 1675791934,
  "states": [
    [
      "a57b26",
      "N452SM  ",
      "United States",
      1675791621,
      1675791621,
      -105.1168,
      39.9103,
      null,
      true,
      0,
      90,
      null,
      null,
      null,
      null,
      false,
      0
    ],
    [
      "aa56da",
      "UAL1986 ",
      "United States",
      1675791933,
      1675791933,
      -122.1349,
      41.1152,
      10972.8,
      false,
      235.33,
      2.51,
      0,
      null,
      11049,
      null,
      false,
      0
    ]
  ]
}

当前Jolt Spec及结果 #

当前使用的Shift转换规则：

[
  {
    "operation": "shift",
    "spec": {
      "states": {
        "*": {
          "*": "[&1]"
        }
      }
    }
  }
]

得到的结果仅提取了嵌套数组，未添加对应键名：

[
  [
    "a57b26",
    "N452SM  ",
    "United States",
    1675791621,
    1675791621,
    -105.1168,
    39.9103,
    null,
    true,
    0,
    90,
    null,
    null,
    null,
    null,
    false,
    0
  ],
  [
    "aa56da",
    "UAL1986 ",
    "United States",
    1675791933,
    1675791933,
    -122.1349,
    41.1152,
    10972.8,
    false,
    235.33,
    2.51,
    0,
    null,
    11049,
    null,
    false,
    0
  ]
]

预期输出 #

[
  {
    "icao24": "a57b26",
    "callsign": "N452SM  ",
    "origin_country": "United States",
    "time_position": 1675791621,
    "last_contact": 1675791621,
    "long": -105.1168,
    "lat": 39.9103,
    "baro_altitude": null,
    "on_ground": true,
    "velocity": 0,
    "true_track": 90,
    "vertical_rate": null,
    "sensors": null,
    "geo_altitude": null,
    "squawk": null,
    "spi": false,
    "position_source": 0
  },
  {
    "icao24": "aa56da",
    "callsign": "UAL1986 ",
    "origin_country": "United States",
    "time_position": 1675791933,
    "last_contact": 1675791933,
    "long": -122.1349,
    "lat": 41.1152,
    "baro_altitude": 10972.8,
    "on_ground": false,
    "velocity": 235.33,
    "true_track": 2.51,
    "vertical_rate": 0,
    "sensors": null,
    "geo_altitude": 11049,
    "squawk": null,
    "spi": false,
    "position_source": 0
  }
]

解决方案：正确的Jolt Spec #

通过在Shift操作中为每个数组索引绑定指定键名，即可完成映射，规则如下：

[
  {
    "operation": "shift",
    "spec": {
      "states": {
        "*": {
          "0": "[&1].icao24",
          "1": "[&1].callsign",
          "2": "[&1].origin_country",
          "3": "[&1].time_position",
          "4": "[&1].last_contact",
          "5": "[&1].long",
          "6": "[&1].lat",
          "7": "[&1].baro_altitude",
          "8": "[&1].on_ground",
          "9": "[&1].velocity",
          "10": "[&1].true_track",
          "11": "[&1].vertical_rate",
          "12": "[&1].sensors",
          "13": "[&1].geo_altitude",
          "14": "[&1].squawk",
          "15": "[&1].spi",
          "16": "[&1].position_source"
        }
      }
    }
  }
]

规则解释 #

• states：定位到输入中的states数组字段
• "*"：遍历states中的每一个子数组（单条状态数据），&1引用当前父级的索引，用来对应结果数组的位置
• 数字键0到16对应子数组的元素索引，将元素映射到指定键名，比如"0": "[&1].icao24"表示把第0个元素赋值给结果数组对应位置对象的icao24字段

内容的提问来源于stack exchange，提问作者james