Apache Arrow IPC流SPMC并发读取的正确性及异常问题问询

核心疑问 #

• 是否支持多个IPC流读取器并发追踪来自单个进程的流写入器？从IPC流的相关描述推测该场景可行，但未在官方文档中找到明确确认，仅在C++写入进程的源码中看到std::mutex用于保护并发写入操作。

实际异常场景 #

环境配置 #

• C++生产者：Arrow 10.0.0版本，通过arrow::ipc::MakeStreamWriter(arrow::io::FileOutputStream)接口写入数据
• Python消费者：Arrow 12版本，通过pa.ipc.open_stream(f)接口读取流数据

异常表现 #

多数情况下读写流程运行正常，但偶尔读取器会遭遇损坏的记录批次，抛出如下错误：

_logger.exception(f'error writing batch {table_from_batches.to_pandas()}')
File "pyarrow/array.pxi", line 837, in pyarrow.lib._PandasConvertible.to_pandas
File "pyarrow/table.pxi", line 2448, in pyarrow.lib.RecordBatch._to_pandas
File "pyarrow/table.pxi", line 4114, in pyarrow.lib.Table._to_pandas
File "../.venv/lib/python3.8/site-packages/pyarrow/pandas_compat.py", line 820, in table_to_blockmanager
    blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
File "../.venv/lib/python3.8/site-packages/pyarrow/pandas_compat.py", line 1168, in _table_to_blocks
    result = pa.lib.table_to_blocks(options, block_table, categories,
File "pyarrow/table.pxi", line 2771, in pyarrow.lib.table_to_blocks
File "pyarrow/error.pxi", line 127, in pyarrow.lib.check_status
pyarrow.lib.ArrowIndexError: Index 80 out of bounds

重新读取流可获取正确数据，推测异常由并发竞态条件导致。

内容的提问来源于stack exchange，提问作者kdkavanagh