如何用Jolt Transform实现多源字段映射并保留其他字段？

问题背景 #

处理JSON输入时，不同消息的字段路径存在差异，需要将多路径的同一含义字段映射到统一目标路径，同时保留原有其他字段：

• 部门ID/名称可能来自employee.department.id、employee.department.org.id或employee.organization.department.id，最终要统一映射到employee.department.organization.id和employee.department.organization.name
• sourceSystem字段需要从sourceSystem、originNotification.domain或receivedMetadata.domain任一路径取值

使用SHIFT transform会丢失未指定的其他字段，因此选择modify-overwrite-beta，但此前配置无法正确实现需求。

尝试的错误配置 #

[
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "employee.department.id": "@(employee.department.id)",
      "employee.department.id": "@(employee.department.org.id)"
    }
  }
]

输入示例 #

输入示例1 #

{
  "employee": {
    "id": 1,
    "name": "John",
    "sourceSystem":"test",
    "department": {
      "id": 1,
      "name": "management"
    }
  }
}

输入示例2 #

{
  "employee": {
    "id": 1,
    "name": "John",
    "department": {
      "org": {
        "id": 1,
        "name": "management"
      }
    },
    "originNotification":{
       "domain":"test"
    }
  }
}

输入示例3 #

{
  "employee": {
    "id": 1,
    "name": "John",
    "organization": {
      "department": {
        "id": 1,
        "name": "management"
      }
    },
    "receivedMetadata":{
        "domain":"test"
    }
  }
}

期望输出 #

{
  "employee": {
    "id": 1,
    "name": "John",
    "department": {
        "organization":{
            "id": 1,
            "name": "management"
        }
    },
    "sourceSystem":"test"
  }
}

正确配置方案 #

[
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      // 按优先级取sourceSystem的有效值，路径存在则覆盖
      "employee.sourceSystem": "@(employee.sourceSystem)",
      "employee.sourceSystem": "@(employee.originNotification.domain)",
      "employee.sourceSystem": "@(employee.receivedMetadata.domain)",
      // 构建统一的部门组织结构，从多路径提取id和name
      "employee.department.organization": {
        "id": "@(employee.department.id)",
        "id": "@(employee.department.org.id)",
        "id": "@(employee.organization.department.id)",
        "name": "@(employee.department.name)",
        "name": "@(employee.department.org.name)",
        "name": "@(employee.organization.department.name)"
      },
      // 清理冗余的原始字段
      "employee.department.id": "",
      "employee.department.org": "",
      "employee.organization": "",
      "employee.originNotification": "",
      "employee.receivedMetadata": ""
    }
  }
]

配置说明 #

1. 多源取值优先级：同一字段的多次赋值会按顺序覆盖，仅保留最后一个存在的有效值，因此将优先级高的路径放在前面，后续路径作为降级备选。
2. 统一目标结构：直接创建employee.department.organization目标结构，从所有可能的原始路径中提取对应字段值，确保输出结构一致。
3. 冗余字段清理：通过将冗余路径设为空字符串""，移除输入中不需要的原始字段，避免输出冗余内容。
4. 保留原有字段：modify-overwrite-beta仅修改spec中指定的字段，未提及的employee.id、employee.name等字段会自动保留，不会丢失。

内容的提问来源于stack exchange，提问作者Scorpy