You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

如何重新读取LZO压缩的Pandas HDF5文件?RHEL8适配问题

解决RHEL-8下无法读取LZO压缩的Pandas HDF5文件问题

我将数据存储为Pandas HDF5文件,部分使用lzo压缩,部分使用blosc:zstd压缩。在RHEL-7系统中可正常读取两种文件,但升级到RHEL-8后,仅能读取blosc:zstd压缩的文件,读取lzo压缩文件时出现以下报错:

...
    return pd.read_hdf(...)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../pyenv/versions/3.13.5_1/lib/python3.13/site-packages/pandas/io/pytables.py", line 457, in read_hdf
    return store.select(
           ~~~~~~~~~~~~^
        key,
        ^^^^
    ...<6 lines>...
        auto_close=auto_close,
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File ".../pyenv/versions/3.13.5_1/lib/python3.13/site-packages/pandas/io/pytables.py", line 911, in select
    return it.get_result()
           ~~~~~~~~~~~~~^^
  File ".../pyenv/versions/3.13.5_1/lib/python3.13/site-packages/pandas/io/pytables.py", line 2034, in get_result
    results = self.func(self.start, self.stop, where)
  File ".../pyenv/versions/3.13.5_1/lib/python3.13/site-packages/pandas/io/pytables.py", line 895, in func
    return s.read(start=_start, stop=_stop, where=_where, columns=columns)
           ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../pyenv/versions/3.13.5_1/lib/python3.13/site-packages/pandas/io/pytables.py", line 4913, in read
    df = super().read(where=where, columns=columns, start=start, stop=stop)
  File ".../pyenv/versions/3.13.5_1/lib/python3.13/site-packages/pandas/io/pytables.py", line 4662, in read
    result = self._read_axes(where=where, start=start, stop=stop)
  File ".../pyenv/versions/3.13.5_1/lib/python3.13/site-packages/pandas/io/pytables.py", line 3845, in _read_axes
    values = selection.select()
  File ".../pyenv/versions/3.13.5_1/lib/python3.13/site-packages/pandas/io/pytables.py", line 5447, in select
    return self.table.table.read_where(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self.condition.format(), start=self.start, stop=self.stop
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File ".../pyenv/versions/3.13.5_1/lib/python3.13/site-packages/tables/table.py", line 1644, in read_where
    p.nrow for p in self._where(condition, condvars, start, stop, step)
                    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "tables/tableextension.pyx", line 1047, in tables.tableextension.Row.__next__
  File "tables/tableextension.pyx", line 1094, in tables.tableextension.Row._Row__next__indexed
  File "tables/tableextension.pyx", line 746, in tables.tableextension.Table._read_chunk
tables.exceptions.HDF5ExtError: HDF5 error back trace

  File "H5D.c", line 1061, in H5Dread
    can't synchronously read data
  File "H5D.c", line 1008, in H5D__read_api_common
    can't read data
  File "H5VLcallback.c", line 2092, in H5VL_dataset_read_direct
    dataset read failed
  File "H5VLcallback.c", line 2048, in H5VL__dataset_read
    dataset read failed
  File "H5VLnative_dataset.c", line 373, in H5VL__native_dataset_read
    can't read data
  File "H5Dio.c", line 401, in H5D__read
    can't read data
  File "H5Dchunk.c", line 2919, in H5D__chunk_read
    unable to read raw data chunk
  File "H5Dchunk.c", line 4548, in H5D__chunk_lock
    data pipeline read failed
  File "H5Z.c", line 1354, in H5Z_pipeline
    required filter 'lzo' is not registered
  File "H5PLint.c", line 265, in H5PL_load
    can't find plugin. Check either HDF5_VOL_CONNECTOR, HDF5_PLUGIN_PATH, default location, or path set by H5PLxxx functions
  File "H5PLpath.c", line 804, in H5PL__find_plugin_in_path_table
    search in path /usr/local/hdf5/lib/plugin encountered an error
  File "H5PLpath.c", line 856, in H5PL__find_plugin_in_path
    can't open directory (/usr/local/hdf5/lib/plugin). Please verify its existence

End of HDF5 error back trace

Problems reading chunk records.

已尝试的操作:

  • 安装lzo和lzo-devel包
  • 重新编译Python
  • 重装所有Python依赖包
  • 从源码编译HDF5

更新信息:PyTables v3.10.1的wheel包不再包含lzo,Linux版conda包也移除了lzo支持。


解决方案

1. 从源码编译PyTables(指定LZO支持)

选择PyTables 3.9.2及以下版本(3.10+版本默认移除了LZO打包),按以下步骤操作:

  1. 安装系统依赖:
sudo yum install lzo lzo-devel hdf5-devel python3-devel gcc gcc-c++
  1. 下载指定版本的PyTables源码:
git clone --branch v3.9.2 https://github.com/PyTables/PyTables.git
cd PyTables
  1. 编译并安装,启用LZO:
python setup.py build --lzo
python setup.py install

2. 配置HDF5插件路径(可选)

如果编译后仍提示找不到LZO插件,手动设置HDF5插件环境变量:

export HDF5_PLUGIN_PATH=/usr/lib64/hdf5/plugin/

(若你的LZO插件位于其他路径,替换为实际路径)

3. 验证LZO支持

安装完成后,运行以下代码验证:

import tables
print(tables.which_libs())

输出中需包含lzo: YES,表示LZO支持已启用。


内容的提问来源于stack exchange,提问作者S.V

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.06.12 10:25:23