使用Xarray在Python中打开.nc文件时出现HDF5错误

问题场景 #

用xarray的open_mfdataset批量加载MERRA-2格式的.nc文件，执行print(data.OMEGA.values)时，每个文件都会触发HDF5属性无法打开的错误。虽然最终能得到结果，但耗时远高于预期。

代码示例 #

import xarray as xr

data = xr.open_mfdataset('/path/to/my/data/*.nc')
print(data.OMEGA.values)

错误日志 #

HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 5:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #005: H5Oattribute.c line 476 in H5O__attr_open_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #006: H5Adense.c line 394 in H5A__dense_open(): can't locate attribute in name index
    major: Attribute
    minor: Object not found

解决方案 #

MERRA-2的HDF5文件常存在非标准属性或索引问题，xarray默认的读取逻辑会遍历所有属性导致错误和性能损耗，可尝试以下方法：

1. 仅加载目标变量 #

在open_mfdataset中指定data_vars=['OMEGA']，只读取需要的变量，避免遍历无关属性：

import xarray as xr

data = xr.open_mfdataset('/path/to/my/data/*.nc', data_vars=['OMEGA'])
print(data.OMEGA.values)

2. 指定引擎并跳过冗余解析 #

使用netcdf4引擎，同时关闭mask_and_scale和decode_cf（无需CF元数据时），减少属性读取操作：

data = xr.open_mfdataset('/path/to/my/data/*.nc', 
                         engine='netcdf4',
                         mask_and_scale=False,
                         decode_cf=False,
                         data_vars=['OMEGA'])

3. 分块读取数据 #

针对大文件，按维度分块加载，降低单次读取的内存和IO压力：

data = xr.open_mfdataset('/path/to/my/data/*.nc', 
                         data_vars=['OMEGA'],
                         chunks={'time': 1, 'lat': 30, 'lon': 30})  # 根据实际维度调整块大小
print(data.OMEGA.values)

4. 逐个文件处理 #

如果批量读取仍有问题，循环逐个打开文件并拼接，手动规避属性错误：

import os
import xarray as xr

file_paths = [os.path.join('/path/to/my/data', f) for f in os.listdir('/path/to/my/data') if f.endswith('.nc')]
datasets = []
for path in file_paths:
    ds = xr.open_dataset(path, data_vars=['OMEGA'], engine='netcdf4', mask_and_scale=False)
    datasets.append(ds)
data = xr.concat(datasets, dim='time')
print(data.OMEGA.values)

内容的提问来源于stack exchange，提问作者Logan