使用ChoETL将嵌套JSON转为Parquet时结构扁平化问题求助

问题背景 #

使用ChoETL将嵌套JSON转换为Parquet文件时，默认会自动扁平化嵌套字段（如device.type_id被转为device_type_id），同时时间字段被转换为数值类型，需要保留原JSON的嵌套层级结构并正确处理时间格式。

原始嵌套JSON #

{"time": 1637045320491, "device": {"type_id": 1}, "message": "Test message", "metadata": {"product": {"name": "prodName", "vendor_name": "XYZ"}, "version": "1.0.0", "original_time": "2021-11-16T12:18:40.491893+05:30"}, "tuid": 900201, "cluid": 9002, "activity_id": 1, "severity_id": 4, "cuid": 9}

转换后扁平化的结果 #

{"time":1637045320491,"device_type_id":1,"message":"Test message","metadata_product_name":"prodName","metadata_product_vendor_name":"XYZ","metadata_version":"1.0.0","metadata_original_time":24520491893000,"tuid":900201,"cluid":9002,"activity_id":1,"severity_id":4,"cuid":9}

原转换代码 #

var pqFile = @"D:\Data\" + Guid.NewGuid() + ".parquet";
using (var r = new ChoJSONReader(@"D:\Data\json-dump.json"))
{
    using (var w = new ChoParquetWriter(pqFile))
    {
         w.Write(r);
    }
}

解决方案 #

ChoETL默认启用FlattenNode配置来扁平化嵌套结构，只需禁用该配置，同时可通过强类型映射或动态配置保留嵌套层级，还需指定时间格式避免数值转换问题。

方法一：强类型映射（推荐） #

先定义与JSON结构完全匹配的嵌套类：

public class Device
{
    public int type_id { get; set; }
}

public class Product
{
    public string name { get; set; }
    public string vendor_name { get; set; }
}

public class Metadata
{
    public Product product { get; set; }
    public string version { get; set; }
    public DateTime original_time { get; set; }
}

public class LogEntry
{
    public long time { get; set; }
    public Device device { get; set; }
    public string message { get; set; }
    public Metadata metadata { get; set; }
    public int tuid { get; set; }
    public int cluid { get; set; }
    public int activity_id { get; set; }
    public int severity_id { get; set; }
    public int cuid { get; set; }
}

修改转换代码，禁用扁平化并指定时间格式：

var pqFile = @"D:\Data\" + Guid.NewGuid() + ".parquet";
using (var r = new ChoJSONReader<LogEntry>(@"D:\Data\json-dump.json")
       .Configure(c => c.DateTimeFormat = "yyyy-MM-ddTHH:mm:ss.ffffffzzz"))
{
    using (var w = new ChoParquetWriter<LogEntry>(pqFile)
           .Configure(c => c.FlattenNode = false))
    {
         w.Write(r);
    }
}

方法二：动态配置（无需定义类） #

直接在读写器配置中禁用扁平化，并指定时间格式：

var pqFile = @"D:\Data\" + Guid.NewGuid() + ".parquet";
using (var r = new ChoJSONReader(@"D:\Data\json-dump.json")
       .Configure(c => 
       {
           c.FlattenNode = false;
           c.DateTimeFormat = "yyyy-MM-ddTHH:mm:ss.ffffffzzz";
       }))
{
    using (var w = new ChoParquetWriter(pqFile)
           .Configure(c => c.FlattenNode = false))
    {
         w.Write(r);
    }
}

内容的提问来源于stack exchange，提问作者Anand