使用Jolt Transform将多份扁平JSON转换为嵌套JSON

需求：将多份扁平JSON转换为嵌套JSON格式，按唯一载荷聚合数据，并将站点信息归集到stops数组中。

输入数据 #

[
  {
    "container_id": "DEF_id",
    "haulType": "OL",
    "loadNumber": "DO123345",
    "billOfLading": "DO12345",
    "referenceNumbers": "LoadIDEF",
    "addressLine1": "DEF_address",
    "stopReferenceId": "0004",
    "stopType": "PL",
    "containerNumber": "454545"
  },
  {
    "container_id": "DEF_id",
    "haulType": "OL",
    "loadNumber": "DO123345",
    "billOfLading": "DO12345",
    "referenceNumbers": "LoadIDEF",
    "addressLine1": null,
    "stopReferenceId": "0003",
    "stopType": "PU",
    "containerNumber": "454545"
  },
  {
    "container_id": "ABC_id",
    "haulType": "IL",
    "loadNumber": "BO123345",
    "billOfLading": "BO12345",
    "referenceNumbers": "LoadID",
    "addressLine1": null,
    "stopReferenceId": "0002",
    "stopType": "PL",
    "containerNumber": "232323"
  },
  {
    "container_id": "ABC_id",
    "haulType": "IL",
    "loadNumber": "BO123345",
    "billOfLading": "BO12345",
    "referenceNumbers": "LoadID",
    "addressLine1": "ABC Street",
    "stopReferenceId": "0001",
    "stopType": "PU",
    "containerNumber": "232323"
  }
]

预期输出 #

[
  {
    "load": {
      "container_id": "DEF_id",
      "haulType": [
        "OL"
      ],
      "loadNumber": "DO123345",
      "billOfLading": "DO12345",
      "referenceNumbers": [
        "LoadIDEF"
      ],
      "stops": [
        {
          "addressLine1": "DEF_address",
          "stopReferenceId": "0004",
          "stopType": "PL"
        },
        {
          "addressLine1": null,
          "stopReferenceId": "0003",
          "stopType": "PU"
        }
      ]
    },
    "containerInfo": {
      "containerNumber": "454545"
    }
  },
  {
    "load": {
      "container_id": "ABC_id",
      "haulType": [
        "IL"
      ],
      "loadNumber": "BO123345",
      "billOfLading": "BO12345",
      "referenceNumbers": [
        "LoadID"
      ],
      "stops": [
        {
          "addressLine1": null,
          "stopReferenceId": "0002",
          "stopType": "PL"
        },
        {
          "addressLine1": "ABC Street",
          "stopReferenceId": "0001",
          "stopType": "PU"
        }
      ]
    },
    "containerInfo": {
      "containerNumber": "232323"
    }
  }
]

当前使用的Jolt规格 #

[
  {
    "operation": "shift",
    "spec": {
      "*": {
        "container_id": "@(1,containerNumber).load.&",
        "haulType": "@(1,containerNumber).load.&",
        "loadNumber": "@(1,containerNumber).load.&",
        "billOfLading": "@(1,containerNumber).load.&",
        "referenceNumbers": "@(1,containerNumber).load.&",
        "addressLine1": "@(1,containerNumber).load.stops[&1].&",
        "stopReferenceId": "@(1,containerNumber).load.stops[&1].&",
        "stopType": "@(1,containerNumber).load.stops[&1].&",
        "containerNumber": "@(1,containerNumber).containerInfo.&"
      }
    }
  },
  {
    "operation": "cardinality",
    "spec": {
      "*": {
        "*": {
          "*": "ONE",
          "stops": "MANY"
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": "&",
        "load": {
          "haulType|referenceNumbers": "&1.&[]",
          "*": "&1.&"
        }
      }
    }
  }
]

修正后的Jolt规格 #

[
  {
    "operation": "shift",
    "spec": {
      "*": {
        "container_id": "@(1,containerNumber).load.&",
        "haulType": "@(1,containerNumber).load.&",
        "loadNumber": "@(1,containerNumber).load.&",
        "billOfLading": "@(1,containerNumber).load.&",
        "referenceNumbers": "@(1,containerNumber).load.&",
        "addressLine1": "@(1,containerNumber).load.stops[#2].&",
        "stopReferenceId": "@(1,containerNumber).load.stops[#2].&",
        "stopType": "@(1,containerNumber).load.stops[#2].&",
        "containerNumber": "@(1,containerNumber).containerInfo.&"
      }
    }
  },
  {
    "operation": "cardinality",
    "spec": {
      "*": {
        "load": {
          "container_id": "ONE",
          "haulType": "ONE",
          "loadNumber": "ONE",
          "billOfLading": "ONE",
          "referenceNumbers": "ONE",
          "stops": "MANY"
        },
        "containerInfo": {
          "containerNumber": "ONE"
        }
      }
    }
  },
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "*": {
        "load": {
          "haulType": ["@(1,haulType)"],
          "referenceNumbers": ["@(1,referenceNumbers)"]
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "*": "[]"
    }
  }
]

修正说明 #

1. Shift操作优化：将stops数组的索引从&1改为#2，确保每个原始输入条目对应一个独立的stop对象，避免索引混乱；
2. Cardinality精准配置：明确指定每个字段的基数，避免层级匹配错误，确保聚合后单值字段保留唯一值，stops保留多值数组；
3. 数组格式转换：通过modify-overwrite-beta操作将haulType和referenceNumbers转为预期的数组格式；
4. 最终结构调整：最后一步Shift操作将分组后的键值对结构转换为预期的数组格式。

内容的提问来源于stack exchange，提问作者VamsiKumar_gudala