使用h5py保存10K图像时文件损坏，读取报OSError问题求助

一直用代码将图像保存为h5py文件，此前运行正常，但处理10K张特定图像时，生成的所有h5py文件均损坏；换用其他图像则可正常运行。已用matplotlib绘图确认图像本身无问题，尝试移除压缩参数也无法解决问题。

保存图像代码 #

f, ax = plt.subplots(1, 2, figsize=(20, 20))
ax[0].imshow(img)
ax[0].set_title('Nuova Img')
ax[1].imshow(gt_mask)
ax[1].set_title('Vecchia img')
plt.savefig("my_image.png")
with h5py.File(os.path.join(output_path, output_fn), "w") as data_h5:
    data_h5.create_dataset("image", data=img, compression="szip")

读取文件脚本及报错 #

读取脚本：

# 读取文件的脚本（报错信息如下）
with h5py.File(path1, "r") as data_h5:
    img1 = np.array(data_h5["image"])
    mask1 = np.array(data_h5["mask"])

报错信息：

Traceback (most recent call last):
  File "hdf_reader.py", line 41, in <module>
    main()
  File "hdf_reader.py", line 15, in main
    img1 = np.array(data_h5["image"])
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/Library/Python/3.9/lib/python/site-packages/h5py/_hl/dataset.py", line 1063, in __array__
    self.read_direct(arr)
  File "/Library/Python/3.9/lib/python/site-packages/h5py/_hl/dataset.py", line 1024, in read_direct
    self.id.read(mspace, fspace, dest, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 243, in h5py.h5d.DatasetID.read
  File "h5py/_proxy.pyx", line 112, in h5py._proxy.dset_rw
OSError: Can't read data (can't open directory: /usr/local/hdf5/lib/plugin)

解决方案 #

• 修复HDF5插件目录
  先检查/usr/local/hdf5/lib/plugin目录是否存在，不存在则创建并设置权限：

  mkdir -p /usr/local/hdf5/lib/plugin
  chmod 755 /usr/local/hdf5/lib/plugin
  

  若目录已存在，修复权限：

  chown -R $USER:$USER /usr/local/hdf5/lib/plugin
  chmod -R 755 /usr/local/hdf5/lib/plugin
  

• 跳过插件目录检查
  在读取h5py文件前清空插件路径环境变量，避免加载插件：

  import os
  os.environ["HDF5_PLUGIN_PATH"] = ""
  import h5py
  
  # 执行读取逻辑
  with h5py.File(path1, "r") as data_h5:
      img1 = np.array(data_h5["image"])
      mask1 = np.array(data_h5["mask"])
  

• 更换压缩格式
  改用h5py原生支持的gzip压缩，避免szip对插件的依赖：

  with h5py.File(os.path.join(output_path, output_fn), "w") as data_h5:
      data_h5.create_dataset("image", data=img, compression="gzip", compression_opts=9)
  

• 检查版本兼容性
  查看h5py与HDF5版本是否匹配：

  pip show h5py
  h5dump --version
  

  若版本不兼容，重新安装h5py：

  pip uninstall h5py -y
  pip install h5py --no-binary h5py
  

内容的提问来源于stack exchange，提问作者Uomolepre