C++序列化IPC Buffer在Python正常解析但JavaScript报错排查

问题概述 #

使用C++ Arrow 14.0.1序列化的IPC Buffer，可在Python（Arrow 5）中正常反序列化为DataFrame，但在JavaScript（Arrow 14.0.1 + Node.js 14）中触发元数据字节不匹配错误：

Error fetching market_data data: Error: Expected to read 131072 metadata bytes, but only read 120532.

复现代码 #

C++ 序列化代码 #

arrow::Result<std::shared_ptr<arrow::Buffer>> DataFrame::toBinary(std::vector<std::string> columns,
                                                                  std::optional<std::string> const &index,
                                                                  std::unordered_map<std::string, std::string> const& metadata) const {
    columns = columns.empty() ? this->columnNames() : columns;

    std::shared_ptr<arrow::RecordBatch> array = m_array;
    if (index) {
        array = array->AddColumn(array->num_columns(), arrow::field(*index, m_index->type()), m_index).MoveValueUnsafe();
    }

    std::shared_ptr<arrow::io::BufferOutputStream> output_stream;
    ARROW_ASSIGN_OR_RAISE(output_stream, arrow::io::BufferOutputStream::Create());

    // Create IPC writer
    std::shared_ptr<arrow::ipc::RecordBatchWriter> writer;
    ARROW_ASSIGN_OR_RAISE(writer, arrow::ipc::MakeStreamWriter(output_stream.get(), array->schema()));
    // Write the RecordBatch
    ARROW_RETURN_NOT_OK(writer->WriteRecordBatch(*array, std::make_shared<arrow::KeyValueMetadata>(metadata)));

    // Finalize the writer
    ARROW_RETURN_NOT_OK(writer->Close());

    // Retrieve the buffer
    std::shared_ptr<arrow::Buffer> buffer;
    ARROW_ASSIGN_OR_RAISE(buffer, output_stream->Finish());

    return buffer;
}

Python 反序列化代码 #

import pyarrow as pa
import pyarrow.ipc as ipc

buffer = pa.BufferReader(binary_data)
reader = ipc.open_stream(buffer)
table = reader.read_all()
data_frame = table.to_pandas()
print(data_frame.columns)
print(data_frame)

JavaScript 初始错误反序列化代码 #

const { tableFromIPC } = require('apache-arrow');

const table = tableFromIPC(binary_data);

预期与实际行为 #

• 预期：C++序列化的IPC Buffer可在Python和JavaScript中正常反序列化
• 实际：
  • Python：反序列化正常，可正确输出DataFrame
  • JavaScript：反序列化失败，抛出元数据字节不匹配错误

环境信息 #

C++: Arrow version 14.0.1
Python: Arrow version 5
JavaScript: Arrow version 14.0.1, Node.js 14

解决方案 #

经排查，问题根源在于JavaScript中对二进制数据的处理方式错误：此前直接将字符串格式的二进制数据传入tableFromIPC，导致二进制内容被错误解析。正确的处理方式是先将字符串形式的二进制数据转换为Node.js Buffer：

const { tableFromIPC } = require('apache-arrow');

// 将字符串格式的二进制数据转换为Buffer
const buffer = Buffer.from(stringBuffer, 'binary');
const table = tableFromIPC(buffer);

内容的提问来源于stack exchange，提问作者user26260915