C#写入Apache Arrow文件在Julia/Python读取时的Footer异常问题

问题重现 #

1. C#写入Struct Array到Arrow文件的代码 #

var structField = new StructType(
    new []
    {
        new Field("field1", new StringType(), nullable: false),
        new Field("field2", new Int64Type(), nullable: false)
    });

StringArray stringArray = new StringArray.Builder()
    .AppendRange(new[] {"angel", "bobby", "charlie"})
    .Build();
Int64Array intArray = new Int64Array.Builder()
    .AppendRange(new[] { 1L,2,3 })
    .Build();

StructArray structs = new StructArray(structField, 3, new IArrowArray[] { stringArray, intArray }, ArrowBuffer.Empty, nullCount: 0);

var recordBatch = new Apache.Arrow.RecordBatch.Builder()
    .Append("col1", false, structs)
    .Build();

using (var stream = File.OpenWrite("test.arrow"))
using (var writer = new Apache.Arrow.Ipc.ArrowFileWriter(stream, recordBatch.Schema, true))
{
    await writer.WriteRecordBatchAsync(recordBatch);
    await writer.WriteEndAsync();
}

2. Julia读取时的错误 #

执行代码：

using Arrow
table = Arrow.Table("test.arrow")
@show table
@show table[1]

报错信息：

ERROR: LoadError: MethodError: no method matching iterate(::Nothing)
Closest candidates are:
  iterate(::Union{LinRange, StepRangeLen}) at range.jl:872
  iterate(::Union{LinRange, StepRangeLen}, ::Integer) at range.jl:872
  iterate(::T) where T<:Union{Base.KeySet{<:Any, <:Dict}, Base.ValueIterator{<:Dict}} at dict.jl:712
  ...
Stacktrace:
 [1] getdictionaries!(dictencoded::Dict{Int64, Arrow.Flatbuf.Field}, field::Arrow.Flatbuf.Field)
   @ Arrow ~/.julia/packages/Arrow/P0wVk/src/table.jl:409
 [2] getdictionaries!(dictencoded::Dict{Int64, Arrow.Flatbuf.Field}, field::Arrow.Flatbuf.Field)
   @ Arrow ~/.julia/packages/Arrow/P0wVk/src/table.jl:410
 [3] macro expansion
   @ ~/.julia/packages/Arrow/P0wVk/src/table.jl:339 [inlined]
 [4] macro expansion
   @ ./task.jl:454 [inlined]
 [5] Arrow.Table(blobs::Vector{Arrow.ArrowBlob}; convert::Bool)
   @ Arrow ~/.julia/packages/Arrow/P0wVk/src/table.jl:321
 [6] Table
   @ ~/.julia/packages/Arrow/P0wVk/src/table.jl:295 [inlined]
 [7] #Table#98
   @ ~/.julia/packages/Arrow/P0wVk/src/table.jl:290 [inlined]
 [8] Table (repeats 2 times)
   @ ~/.julia/packages/Arrow/P0wVk/src/table.jl:290 [inlined]
 [9] top-level scope

使用Stream读取也出现类似错误：

stream = Arrow.Stream("test.arrow")
for d in stream
    @show d
end

3. Python读取时的错误 #

执行代码：

import pyarrow.feather as feather
table = feather.read_table("test.arrow")
print(table[0])

报错信息：

OSError: Verification of flatbuffer-encoded Footer failed.

问题原因及解决方法 #

原因分析 #

问题出在StructArray的构造参数上：原代码传递了ArrowBuffer.Empty作为validity buffer，但根据Apache Arrow规范，当字段没有null值（nullCount:0）时，validity buffer应该不存在（即传递null）。传递空的buffer会导致生成的Arrow文件结构不符合规范，进而引发跨语言读取时的Footer验证失败或结构解析错误。

解决代码 #

修改StructArray的创建代码，移除ArrowBuffer.Empty参数（或替换为null）：

// 方法1：省略validityBuffer参数，使用默认值null
StructArray structs = new StructArray(structField, 3, new IArrowArray[] { stringArray, intArray }, nullCount: 0);

// 方法2：显式传递null作为validityBuffer
StructArray structs = new StructArray(structField, 3, new IArrowArray[] { stringArray, intArray }, null, nullCount: 0);

修改后重新生成Arrow文件，即可在Julia和Python中正常读取。

内容的提问来源于stack exchange，提问作者BAR