使用Parquet.Net写入Null值导致Parquet文件不可读的问题

1. 修正文件路径转义错误 #

原代码中"c:\test.parquet"的路径写法存在转义问题，\t会被解析为制表符，导致实际生成的文件位置不符合预期。需修改为：

using (Stream fileStream = System.IO.File.OpenWrite(@"c:\test.parquet"))

或：

using (Stream fileStream = System.IO.File.OpenWrite("c:\\test.parquet"))

2. 显式标记字符串列为可空 #

尽管使用了string?，但显式指定isNullable: true能确保Parquet Schema正确标记该列允许Null值，避免解析时出现兼容性问题：

var fields = new List<DataField>
{
    new DataField<int>("id"),
    new DataField<string>("city", isNullable: true)
};

3. 关闭字典编码 #

部分Parquet查看工具对包含Null值的字典编码列支持不佳，暂时关闭该特性可解决读取失败问题：

ParquetOptions parquetOptions = new ParquetOptions { TreatByteArrayAsString = true, UseDictionaryEncoding = false, UseDeltaBinaryPackedEncoding = false };

修改后的完整代码 #

var fields = new List<DataField>
{
    new DataField<int>("id"),
    new DataField<string>("city", isNullable: true)
};

var schema = new ParquetSchema(fields);

Parquet.Data.DataColumn[] columns = new Parquet.Data.DataColumn[2];
for (int i = 0; i < 2; i++)
{
    Type t = fields[i].ClrType;

    columns[i] = t switch
    {
        Type when typeof(string) == t => new Parquet.Data.DataColumn(fields[i], new string?[] { "London", null }),
        Type when typeof(int) == t => new Parquet.Data.DataColumn(fields[i], new int[] { 1, 2 }),
        _ => throw new NotImplementedException(),
    };
}

using (Stream fileStream = System.IO.File.OpenWrite(@"c:\test.parquet"))
{
    ParquetOptions parquetOptions = new ParquetOptions { TreatByteArrayAsString = true, UseDictionaryEncoding = false, UseDeltaBinaryPackedEncoding = false };

    using (ParquetWriter parquetWriter = await ParquetWriter.CreateAsync(schema, fileStream, parquetOptions))
    {
        parquetWriter.CompressionMethod = CompressionMethod.Gzip;
        parquetWriter.CompressionLevel = System.IO.Compression.CompressionLevel.Optimal;

        using (ParquetRowGroupWriter groupWriter = parquetWriter.CreateRowGroup())
        {
            foreach (var item in columns)
            {
                await groupWriter.WriteColumnAsync(item);
            }
        }
    }
}

额外测试建议 #

如果上述修改后仍无法读取，可尝试将压缩方式改为无压缩（CompressionMethod.None），部分轻量Parquet查看工具对压缩格式的支持可能存在局限性。

内容的提问来源于stack exchange，提问作者Wijnand Beke