使用JOLT移除JSON中的Null属性与空数组问题求助

输入JSON #

{"entityType" : "person","id" : 78,"properties" : {"first_name" : "Ricardo","last_name" : "Bochini","company" : null,"title" : null,"phone_number" : null,"phone_type" : null,"city" : null,"state" : null,"country" : null,"email_home" : null,"email_work" : null,"websites" : null,"employments" : [ {"company_name" : "Disney","start_date" : "08-Apr-2021","end_date" : "","title" : "dev"} ],"tags" : [ "Biotechnology", "Biochemistry", "Western Blotting" ],"willing_to_relocate_" : "true","social_media_addresses" : "linkedin.com/sarasa","areas_of_interest" : [ ],"industry" : [ ],"work_preference_candidate_16838217785051286" : [ ],"availability" : "","job_id" : "4327447002","initial_stage_id" : "6428145002","resume" : {"content" : "JVBERi0xLjUKJcO","type" : "resume","label" : "FSD_CandidatesExport.pdf"},"education_table" : [ {"degree" : "","degree_type" : "","university" : "","start_date" : "","end_date" : "","concentration" : ""} ],"form_search_type" : "Non-executive","form_person_name" : "Luptiqay Dedzomu, Orunaze","form_req_to_link" : "Test Job Integration","adress_type" : "other","website_address_type" : "other","content_type" : "application/pdf","form_background" : "dad","form_technical_experience" : "technichal experience","form_drivers" : "Drivers","form_reasons" : "Reason","form_ideal_role" : "Ideal role","form_areas_of_interest" : [ "Research - Discovery", "Research - Preclinical" ]}}

期望输出 #

移除所有值为null、空字符串或空数组的属性，得到：

{"entityType" : "person","id" : 78,"properties" : {"first_name" : "Ricardo","last_name" : "Bochini","employments" : [ {"company_name" : "Disney","start_date" : "08-Apr-2021","title" : "dev"} ],"tags" : [ "Biotechnology", "Biochemistry", "Western Blotting" ],"willing_to_relocate_" : "true","social_media_addresses" : "linkedin.com/sarasa","availability" : "","job_id" : "4327447002","initial_stage_id" : "6428145002","resume" : {"content" : "JVBERi0xLjUKJcO","type" : "resume","label" : "FSD_CandidatesExport.pdf"},"form_search_type" : "Non-executive","form_person_name" : "Luptiqay Dedzomu, Orunaze","form_req_to_link" : "Test Job Integration","adress_type" : "other","website_address_type" : "other","content_type" : "application/pdf","form_background" : "dad","form_technical_experience" : "technichal experience","form_drivers" : "Drivers","form_reasons" : "Reason","form_ideal_role" : "Ideal role","form_areas_of_interest" : [ "Research - Discovery", "Research - Preclinical" ]}}

尝试过的无效规则 #

[{"operation": "default","spec": {"properties": {"*": "DELETE"}}},{"operation": "shift","spec": {"properties": {"*": {"DELETE": null,"*": {"@1": "&2"}}}}}]

该规则无法正确处理数组类型的属性，导致转换失效。

解决方案 #

使用以下JOLT转换规则，可递归处理所有层级的null、空数组，同时保留空字符串（如availability字段）：

[
  // 第一步：标记需要删除的属性（null、空数组）
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "*": {
        "$": "=notNull",
        "=isEmpty": null,
        "properties": {
          "*": {
            "$": "=notNull",
            "=isEmpty": null,
            "*": {
              "$": "=notNull",
              "=isEmpty": null
            }
          }
        }
      }
    }
  },
  // 第二步：移除标记为null的属性，保留空字符串
  {
    "operation": "shift",
    "spec": {
      "*": "&",
      "properties": {
        "*": {
          "null": null,
          "*": {
            "@1": "&2"
          }
        }
      }
    }
  },
  // 第三步：处理数组内的空属性（如employments里的end_date）
  {
    "operation": "shift",
    "spec": {
      "*": "&",
      "properties": {
        "*": {
          "[]": {
            "*": {
              "": null,
              "*": "&2[&1].&"
            }
          },
          "*": "&1.&"
        }
      }
    }
  }
]

规则说明： #

1. modify-overwrite-beta：利用JOLT内置函数notNull和isEmpty，将null值和空数组标记为待删除的null，空字符串不受影响。
2. 第一个shift操作：移除被标记为null的顶层及properties下的属性，留存有效数据。
3. 第二个shift操作：递归处理数组内的对象，移除其中的空字符串属性，同时保留非空数组和有效对象属性。

内容的提问来源于stack exchange，提问作者Mariano Baudena