如何使用h5py在HDF5中存储定长字符串并解决<U1类型转换报错

报错原因 #

该错误是由于传入的sdata为NumPy Unicode字符串类型（dtype为<U1，对应小端序、长度为1的Unicode字符），而h5py没有提供NumPy Unicode字符串到HDF5内置存储类型的默认转换路径。HDF5原生仅支持基于字节序列的字符串存储，无法直接识别NumPy的Unicode编码字符串数组。

解决方案 #

针对你存储定长短字符串的场景，推荐使用存储效率、读写性能更高的定长字节串转换方案：
将Unicode字符串数组转为定长字节串数组后再写入HDF5，读取时再转回Unicode即可，修改后代码如下：

import numpy as np
import h5py

sdata = [ np.array(["A", "C", "A", "T", "C", "C", "T", "C"]), np.array(["G", "A", "C", "C", "C", "T", "A", "A"]), np.array(["G", "G", "A", "C", "C", "A", "A", "G"]) ]
# 将Unicode字符串数组转为长度为1的定长字节串数组
sdata = np.array(sdata).astype('S1')

h5File = "test.h5"

with h5py.File(h5File, 'w') as h5data:
    h5data.create_dataset('sequence', data=sdata, compression="lzf", chunks=True, maxshape=(None,sdata.shape[1]))

数据读取时的转换示例：

with h5py.File(h5File, 'r') as h5data:
    # 读取后直接转为Unicode字符串数组
    loaded_data = h5data['sequence'][:].astype('U')

如果你需要直接存储Unicode字符串、不想手动做字节转换，可以在创建数据集时指定h5py专属的字符串dtype，该方案更适合字符串长度不固定的场景：

import numpy as np
import h5py

sdata = [ np.array(["A", "C", "A", "T", "C", "C", "T", "C"]), np.array(["G", "A", "C", "C", "C", "T", "A", "A"]), np.array(["G", "G", "A", "C", "C", "A", "A", "G"]) ]
sdata = np.array(sdata)

h5File = "test.h5"

with h5py.File(h5File, 'w') as h5data:
    # 指定字符串dtype，编码为utf-8
    h5data.create_dataset('sequence', data=sdata, dtype=h5py.string_dtype(encoding='utf-8'), compression="lzf", chunks=True, maxshape=(None,sdata.shape[1]))

内容的提问来源于stack exchange，提问作者ybzhao