如何重新读取LZO压缩的Pandas HDF5文件?RHEL8适配问题
解决RHEL-8下无法读取LZO压缩的Pandas HDF5文件问题
我将数据存储为Pandas HDF5文件,部分使用lzo压缩,部分使用blosc:zstd压缩。在RHEL-7系统中可正常读取两种文件,但升级到RHEL-8后,仅能读取blosc:zstd压缩的文件,读取lzo压缩文件时出现以下报错:
... return pd.read_hdf(...) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ".../pyenv/versions/3.13.5_1/lib/python3.13/site-packages/pandas/io/pytables.py", line 457, in read_hdf return store.select( ~~~~~~~~~~~~^ key, ^^^^ ...<6 lines>... auto_close=auto_close, ^^^^^^^^^^^^^^^^^^^^^^ ) ^ File ".../pyenv/versions/3.13.5_1/lib/python3.13/site-packages/pandas/io/pytables.py", line 911, in select return it.get_result() ~~~~~~~~~~~~~^^ File ".../pyenv/versions/3.13.5_1/lib/python3.13/site-packages/pandas/io/pytables.py", line 2034, in get_result results = self.func(self.start, self.stop, where) File ".../pyenv/versions/3.13.5_1/lib/python3.13/site-packages/pandas/io/pytables.py", line 895, in func return s.read(start=_start, stop=_stop, where=_where, columns=columns) ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ".../pyenv/versions/3.13.5_1/lib/python3.13/site-packages/pandas/io/pytables.py", line 4913, in read df = super().read(where=where, columns=columns, start=start, stop=stop) File ".../pyenv/versions/3.13.5_1/lib/python3.13/site-packages/pandas/io/pytables.py", line 4662, in read result = self._read_axes(where=where, start=start, stop=stop) File ".../pyenv/versions/3.13.5_1/lib/python3.13/site-packages/pandas/io/pytables.py", line 3845, in _read_axes values = selection.select() File ".../pyenv/versions/3.13.5_1/lib/python3.13/site-packages/pandas/io/pytables.py", line 5447, in select return self.table.table.read_where( ~~~~~~~~~~~~~~~~~~~~~~~~~~~^ self.condition.format(), start=self.start, stop=self.stop ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File ".../pyenv/versions/3.13.5_1/lib/python3.13/site-packages/tables/table.py", line 1644, in read_where p.nrow for p in self._where(condition, condvars, start, stop, step) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "tables/tableextension.pyx", line 1047, in tables.tableextension.Row.__next__ File "tables/tableextension.pyx", line 1094, in tables.tableextension.Row._Row__next__indexed File "tables/tableextension.pyx", line 746, in tables.tableextension.Table._read_chunk tables.exceptions.HDF5ExtError: HDF5 error back trace File "H5D.c", line 1061, in H5Dread can't synchronously read data File "H5D.c", line 1008, in H5D__read_api_common can't read data File "H5VLcallback.c", line 2092, in H5VL_dataset_read_direct dataset read failed File "H5VLcallback.c", line 2048, in H5VL__dataset_read dataset read failed File "H5VLnative_dataset.c", line 373, in H5VL__native_dataset_read can't read data File "H5Dio.c", line 401, in H5D__read can't read data File "H5Dchunk.c", line 2919, in H5D__chunk_read unable to read raw data chunk File "H5Dchunk.c", line 4548, in H5D__chunk_lock data pipeline read failed File "H5Z.c", line 1354, in H5Z_pipeline required filter 'lzo' is not registered File "H5PLint.c", line 265, in H5PL_load can't find plugin. Check either HDF5_VOL_CONNECTOR, HDF5_PLUGIN_PATH, default location, or path set by H5PLxxx functions File "H5PLpath.c", line 804, in H5PL__find_plugin_in_path_table search in path /usr/local/hdf5/lib/plugin encountered an error File "H5PLpath.c", line 856, in H5PL__find_plugin_in_path can't open directory (/usr/local/hdf5/lib/plugin). Please verify its existence End of HDF5 error back trace Problems reading chunk records.
已尝试的操作:
- 安装lzo和lzo-devel包
- 重新编译Python
- 重装所有Python依赖包
- 从源码编译HDF5
更新信息:PyTables v3.10.1的wheel包不再包含lzo,Linux版conda包也移除了lzo支持。
解决方案
1. 从源码编译PyTables(指定LZO支持)
选择PyTables 3.9.2及以下版本(3.10+版本默认移除了LZO打包),按以下步骤操作:
- 安装系统依赖:
sudo yum install lzo lzo-devel hdf5-devel python3-devel gcc gcc-c++
- 下载指定版本的PyTables源码:
git clone --branch v3.9.2 https://github.com/PyTables/PyTables.git cd PyTables
- 编译并安装,启用LZO:
python setup.py build --lzo python setup.py install
2. 配置HDF5插件路径(可选)
如果编译后仍提示找不到LZO插件,手动设置HDF5插件环境变量:
export HDF5_PLUGIN_PATH=/usr/lib64/hdf5/plugin/
(若你的LZO插件位于其他路径,替换为实际路径)
3. 验证LZO支持
安装完成后,运行以下代码验证:
import tables print(tables.which_libs())
输出中需包含lzo: YES,表示LZO支持已启用。
内容的提问来源于stack exchange,提问作者S.V
相关产品推荐
相关产品推荐

