在NiFi中如何编写JOLT Spec移除仅含空字符串的数组

问题说明 #

需要清理guests对象里值为仅含空字符串的数组字段（heather、brad、thomas、jane），同时保留其他有效字段和原有JSON结构。

输入JSON #

{
  "guests" : {
    "amy" : "amy",
    "theresa" : [ "Chef" ],
    "jim" : [ "Bear Specialist" ],
    "john" : [ "Tour Guide" ],
    "heather" : [ "" ],
    "brad" : [ "" ],
    "thomas" : [ "" ],
    "jane" : [ "" ]
  },
  "dataSchema" : "dataSchema",
  "data" : {
    "key1" : "value1",
    "key2" : {
      "key3" : "value3"
    },
    "label" : "dummyLabel",
    "campingEquipment" : {
      "someKey" : "someValue"
    }
  },
  "mountainRange" : {
    "elevation" : "8000",
    "temperature" : "50"
  }
}

期望输出JSON #

{
  "guests" : {
    "amy" : "amy",
    "theresa" : [ "Chef" ],
    "jim" : [ "Bear Specialist" ],
    "john" : [ "Tour Guide" ]
  },
  "dataSchema" : "dataSchema",
  "data" : {
    "key1" : "value1",
    "key2" : {
      "key3" : "value3"
    },
    "label" : "dummyLabel",
    "campingEquipment" : {
      "someKey" : "someValue"
    }
  },
  "mountainRange" : {
    "elevation" : "8000",
    "temperature" : "50"
  }
}

原方案问题 #

之前的Spec错误删除了所有数组类型字段，原因是：

• recursivelySquashNulls仅处理null值，对空字符串无效
• 后续remove操作直接匹配所有指定字段，未做条件判断，导致有效数组字段也被误删

错误的JOLT Spec：

[
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "guests": {
        "theresa": "=recursivelySquashNulls",
        "jim": "=recursivelySquashNulls",
        "john": "=recursivelySquashNulls",
        "heather": "=recursivelySquashNulls",
        "brad": "=recursivelySquashNulls",
        "thomas": "=recursivelySquashNulls",
        "jane": "=recursivelySquashNulls"
      }
    }
  },
  {
    "operation": "remove",
    "spec": {
      "guests": {
        "theresa": "",
        "jim": "",
        "john": "",
        "heather": "",
        "brad": "",
        "thomas": "",
        "jane": ""
      }
    }
  }
]

正确的JOLT Spec #

使用shift操作结合条件筛选，精准保留有效字段：

[
  {
    "operation": "shift",
    "spec": {
      "guests": {
        "*": {
          // 筛选出数组第一个元素不为空字符串的字段
          "[?(@[0] != '')]": "&2.&",
          // 保留非数组类型的字段（如amy的字符串值）
          "*": "&2.&"
        }
      },
      // 其他顶级字段原样复制
      "*": "&"
    }
  }
]

逻辑解释 #

1. shift操作：JOLT中用于数据重排、筛选的核心操作，支持条件判断
2. guests.*：遍历guests下的所有子字段
3. 条件表达式[?(@[0] != '')]：判断当前字段的数组第一个元素是否为空字符串，满足条件则通过&2.&保留原路径（&2代表父级的父级即guests，&代表当前字段名）
4. "*": "&2.&"：专门处理非数组类型的字段，确保这类字段不会被遗漏
5. "*": "&"：将dataSchema、data、mountainRange等其他顶级字段原样复制到结果中

内容的提问来源于stack exchange，提问作者Ross