Jolt数组扁平化引入多余null元素问题求助

输入JSON数据 #

{
  "changes": [
    {
      "l1_Entity_Id": 7004,
      "l1_Entity_Nm": "Academic Administration2",
      "visibility": false,
      "L2_Entities": [
        {
          "L2_Entity_Id": 8003,
          "L2_Entity_Nm": "Desktop Software2",
          "visibility": 1,
          "primary_triage": "EAAS",
          "markham_triage": "MK_Team1",
          "Faculty_Support": [
            "Faculty1",
            "Faculty2",
            "Faculty4"
          ],
          "hasReference": true
        }
      ]
    },
    {
      "l1_Entity_Id": 7002,
      "l1_Entity_Nm": "Telephony5",
      "visibility": false,
      "L2_Entities": [
        {
          "L2_Entity_Id": 8005,
          "L2_Entity_Nm": "Automatic Call Distribution3",
          "visibility": 1,
          "primary_triage": "Telecom",
          "markham_triage": null,
          "Faculty_Support": [
            "Faculty3",
            "Faculty4"
          ],
          "hasReference": true
        },
        {
          "L2_Entity_Id": 8004,
          "L2_Entity_Nm": "Phone",
          "visibility": 1,
          "primary_triage": null,
          "markham_triage": null,
          "Faculty_Support": [
            "Faculty1"
          ],
          "hasReference": true
        }
      ]
    }
  ],
  "methodName": "generateEntitiesChgReport"
}

期望输出JSON #

[
    {
        "l1_Entity_Nm": "Academic Administration2",
        "L1_visibility": false,
        "L2_Entity_Id": 8003,
        "L2_Entity_Nm": "Desktop Software2",
        "aggregateNm": "Academic Administration2>Desktop Software2",
        "L2_visibility": 1,
        "primary_triage": "EAAS",
        "isMarkham": true,
        "markham_triage": "MK_Team1",
        "Faculty_Support": [
            "Faculty1",
            "Faculty2",
            "Faculty4"
        ]
    },
    {
        "l1_Entity_Nm": "Telephony5",
        "L1_visibility": false,
        "L2_Entity_Id": 8005,
        "L2_Entity_Nm": "Automatic Call Distribution3",
        "aggregateNm": "Telephony5>Automatic Call Distribution3",
        "L2_visibility": 1,
        "primary_triage": "Telecom",
        "isMarkham": false,
        "markham_triage": null,
        "Faculty_Support": [
            "Faculty3",
            "Faculty4"
        ]
    },
    {
        "l1_Entity_Nm": "Telephony5",
        "L1_visibility": false,
        "L2_Entity_Id": 8004,
        "L2_Entity_Nm": "Phone",
        "aggregateNm": "Telephony5>Phone",
        "L2_visibility": 1,
        "primary_triage": null,
        "isMarkham": false,
        "markham_triage": null,
        "Faculty_Support": [
            "Faculty1"
        ]
    }
]

现有Jolt转换规则（存在问题） #

[
  {
    "operation": "shift",
    "spec": {
      "changes": {
        "*": {
          "L2_Entities": {
            "*": {
              "@": "&[&3]",
              "@(2,l1_Entity_Id)": "&[&3].l1_Entity_Id",
              "@(2,l1_Entity_Nm)": "&[&3].l1_Entity_Nm",
              "@(2,l1_visibility)": "&[&3].l1_visibility"
            }
          }
        }
      }
    }
},
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": "[]"
      }
    }
}
]

当前错误输出（含多余null） #

[ {
  "L2_Entity_Id" : 8003,
  "L2_Entity_Nm" : "Desktop Software2",
  "visibility" : 1,
  "primary_triage" : "EAAS",
  "markham_triage" : "MK_Team1",
  "Faculty_Support" : [ "Faculty1", "Faculty2", "Faculty4" ],
  "hasReference" : true,
  "l1_Entity_Id" : 7004,
  "l1_Entity_Nm" : "Academic Administration2"
}, {
  "L2_Entity_Id" : 8005,
  "L2_Entity_Nm" : "Automatic Call Distribution3",
  "visibility" : 1,
  "primary_triage" : "Telecom",
  "markham_triage" : null,
  "Faculty_Support" : [ "Faculty3", "Faculty4" ],
  "hasReference" : true,
  "l1_Entity_Id" : 7002,
  "l1_Entity_Nm" : "Telephony5"
}, null, {
  "L2_Entity_Id" : 8004,
  "L2_Entity_Nm" : "Phone",
  "visibility" : 1,
  "primary_triage" : null,
  "markham_triage" : null,
  "Faculty_Support" : [ "Faculty1" ],
  "hasReference" : true,
  "l1_Entity_Id" : 7002,
  "l1_Entity_Nm" : "Telephony5"
} ]

问题分析与修正后的Jolt规则 #

问题原因 #

1. 现有规则中@(2,l1_visibility)是错误路径，原输入L1层级的字段名为visibility，而非l1_visibility，导致该字段无法被正确提取，间接引发null元素生成。
2. 分组式的shift操作（&[&3]）会按原changes数组的下标对L2元素分组，后续的展开操作处理不当产生空值。
3. 未处理期望输出中要求的aggregateNm、isMarkham等衍生字段，也未移除不需要的hasReference、l1_Entity_Id字段。

修正后的Jolt Spec #

[
  // 第一步：提取并重组基础字段，直接将L2元素放入根数组
  {
    "operation": "shift",
    "spec": {
      "changes": {
        "*": {
          "L2_Entities": {
            "*": {
              "@(2,l1_Entity_Nm)": "[&1].l1_Entity_Nm",
              "@(2,visibility)": "[&1].L1_visibility",
              "L2_Entity_Id": "[&1].L2_Entity_Id",
              "L2_Entity_Nm": "[&1].L2_Entity_Nm",
              "visibility": "[&1].L2_visibility",
              "primary_triage": "[&1].primary_triage",
              "markham_triage": "[&1].markham_triage",
              "Faculty_Support": "[&1].Faculty_Support"
            }
          }
        }
      }
    }
  },
  // 第二步：展开数组，移除层级结构
  {
    "operation": "shift",
    "spec": {
      "*": "[]"
    }
  },
  // 第三步：生成衍生字段aggregateNm和isMarkham
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "*": {
        "aggregateNm": "=concat(@(1,l1_Entity_Nm), '>', @(1,L2_Entity_Nm))",
        "isMarkham": "=notNull(@(1,markham_triage))"
      }
    }
  },
  // 第四步：移除不需要的字段
  {
    "operation": "remove",
    "spec": {
      "*": {
        "hasReference": "",
        "l1_Entity_Id": ""
      }
    }
  }
]

说明 #

1. Shift操作修正：直接使用[&1]将每个L2元素映射到根数组的连续位置，避免分组导致的空值；修正字段引用路径，正确提取L1的visibility并命名为L1_visibility，同时将L2的visibility重命名为L2_visibility。
2. Modify操作：通过concat生成aggregateNm字段；通过notNull判断markham_triage是否非空，生成isMarkham布尔值。
3. Remove操作：移除期望输出中不存在的hasReference、l1_Entity_Id字段。

内容的提问来源于stack exchange，提问作者user22174139