Vaex读取TSV导出Parquet后PyArrow无法读取的问题求助

环境版本信息 #

>>> vaex.__version__
{'vaex': '4.12.0',
 'vaex-core': '4.12.0',
 'vaex-viz': '0.5.3',
 'vaex-hdf5': '0.12.3',
 'vaex-server': '0.8.1',
 'vaex-astro': '0.9.1',
 'vaex-jupyter': '0.8.0',
 'vaex-ml': '0.18.0'}

>>> pyarrow.__version__
8.0.0

测试文件生成 #

通过以下命令生成测试TSV文件s2t.tsv：

$ printf "test-1\nfoobar\ntest-1\nfoobar\ntest-1\nfoobar\ntest-1\nfoobar\n" > s
$ printf "1-best\npoo bear\n1-best\npoo bear\n1-best\npoo bear\n1-best\npoo bear\n" > t
$ paste s t > s2t.tsv

文件内容：

test-1  1-best
foobar  poo bear
test-1  1-best
foobar  poo bear
test-1  1-best
foobar  poo bear
test-1  1-best
foobar  poo bear

问题复现代码 #

执行以下代码导出并读取时出现错误：

import vaex
import pyarrow as pa

df = vaex.from_csv('s2t.tsv', sep='\t', header=None)
df.export_arrow('s2t.parquet')

pa.parquet.read_table('s2t.parquet')

报错信息 #

---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
/tmp/ipykernel_17/3649263967.py in <module>
      1 import pyarrow as pa
      2 
----> 3 pa.parquet.read_table('s2t.parquet')

/opt/conda/lib/python3.7/site-packages/pyarrow/parquet/__init__.py in read_table(source, columns, use_threads, metadata, schema, use_pandas_metadata, memory_map, read_dictionary, filesystem, filters, buffer_size, partitioning, use_legacy_dataset, ignore_prefixes, pre_buffer, coerce_int96_timestamp_unit, decryption_properties)
   2746                 ignore_prefixes=ignore_prefixes,
   2747                 pre_buffer=pre_buffer,
-> 2748                 coerce_int96_timestamp_unit=coerce_int96_timestamp_unit
   2749             )
   2750         except ImportError:

/opt/conda/lib/python3.7/site-packages/pyarrow/parquet/__init__.py in __init__(self, path_or_paths, filesystem, filters, partitioning, read_dictionary, buffer_size, memory_map, ignore_prefixes, pre_buffer, coerce_int96_timestamp_unit, schema, decryption_properties, **kwargs)
   2338 
   2339             self._dataset = ds.FileSystemDataset(
-> 2340                 [fragment], schema=schema or fragment.physical_schema,
   2341                 format=parquet_format,
   2342                 filesystem=fragment.filesystem

/opt/conda/lib/python3.7/site-packages/pyarrow/_dataset.pyx in pyarrow._dataset.Fragment.physical_schema.__get__()

/opt/conda/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

/opt/conda/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Could not open Parquet input source 's2t.parquet': Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

解决方案 #

问题根源是**df.export_arrow()生成的是Arrow IPC格式文件，并非Parquet格式**，文件名使用.parquet后缀会造成格式混淆。提供两种解决方式：

方式一：直接导出Parquet格式 #

使用Vaex专门的export_parquet()方法生成标准Parquet文件，配合pyarrow的Parquet读取方法：

import vaex
import pyarrow as pa

df = vaex.from_csv('s2t.tsv', sep='\t', header=None)
df.export_parquet('s2t.parquet')  # 替换为export_parquet

pa.parquet.read_table('s2t.parquet')  # 可正常读取

方式二：保留Arrow IPC格式读取 #

如果需要使用Arrow原生格式，修改文件后缀为.arrow或.feather，并使用pyarrow的IPC读取方法：

import vaex
import pyarrow as pa

df = vaex.from_csv('s2t.tsv', sep='\t', header=None)
df.export_arrow('s2t.arrow')  # 使用正确的Arrow格式后缀

pa.ipc.read_table('s2t.arrow')  # 用ipc模块读取

内容的提问来源于stack exchange，提问作者alvas