写入NetCDF失败求助：单一气候模型数据导出异常

处理气候模型集合平均数据导出时，其他模型数据均可正常写入NetCDF，但某一模型数据执行导出时失败，不过绘图验证数据显示正常。相关xarray数据集信息及报错如下：

数据集信息 #

<xarray.Dataset>
Dimensions:             (i: 360, j: 300, time: 1032, vertices: 4)
Coordinates:
  * time                (time) datetime64[ns] 2015-01-16T12:00:00 ... 2100-12...
  * j                   (j) int32 0 1 2 3 4 5 6 ... 293 294 295 296 297 298 299
  * i                   (i) int32 0 1 2 3 4 5 6 ... 353 354 355 356 357 358 359
    latitude            (j, i) float64 dask.array<chunksize=(300, 360), meta=np.ndarray>
    longitude           (j, i) float64 dask.array<chunksize=(300, 360), meta=np.ndarray>
Dimensions without coordinates: vertices
Data variables:
    vertices_latitude   (j, i, vertices) float64 dask.array<chunksize=(300, 360, 4), meta=np.ndarray>
    vertices_longitude  (j, i, vertices) float64 dask.array<chunksize=(300, 360, 4), meta=np.ndarray>
    sithick             (time, j, i) float32 dask.array<chunksize=(1032, 300, 360), meta=np.ndarray>

报错信息 #

ds_emean.to_netcdf(path=file_out,mode='w',format='NETCDF4',compute=True)
HDF5-DIAG: Error detected in HDF5 (1.12.0) thread 123145549598720:
  #000: H5Dio.c line 192 in H5Dread(): can't read data
    major: Dataset
    minor: Read failed
  #001: H5VLcallback.c line 2080 in H5VL_dataset_read(): dataset read failed
    major: Virtual Object Layer
    minor: Read failed
  #002: H5VLcallback.c line 2046 in H5VL__dataset_read(): dataset read failed
    major: Virtual Object Layer
    minor: Read failed
  #003: H5VLnative_dataset.c line 167 in H5VL__native_dataset_read(): can't read data
    major: Dataset
    minor: Read failed
  #004: H5Dio.c line 567 in H5D__read(): can't read data
    major: Dataset
    minor: Read failed
  #005: H5Dchunk.c line 2594 in H5D__chunk_read(): unable to read raw data chunk
    major: Low-level I/O
    minor: Read failed
  #006: H5Dchunk.c line 3957 in H5D__chunk_lock(): data pipeline read failed
    major: Dataset
    minor: Filter operation failed
  #007: H5Z.c line 1336 in H5Z_pipeline(): filter returned failure during read
    major: Data filters
    minor: Read failed
  #008: H5Zdeflate.c line 123 in H5Z_filter_deflate(): inflate() failed
    major: Data filters
    minor: Unable to initialize object
Traceback (most recent call last):

可行解决方法 #

• 检查原始文件完整性：报错核心是HDF5解压失败，大概率是该模型的原始输入文件损坏。用h5debug工具检查文件，或重新获取原始数据。
• 调整Dask分块：当前sithick时间维度全量分块（1032个时间步），读写时易引发内存问题。拆分时间分块，比如：

  ds_emean = ds_emean.chunk({'time': 12})
  

  重新计算后再尝试导出。
• 关闭压缩导出：导出时禁用NetCDF的压缩功能，避免解压/压缩环节出错：

  encoding = {var: {'zlib': False} for var in ds_emean.data_vars}
  ds_emean.to_netcdf(path=file_out, mode='w', format='NETCDF4', compute=True, encoding=encoding)
  

• 更新依赖库：HDF5 1.12.0存在已知兼容性问题，更新相关库到最新稳定版：

  pip install --upgrade h5py xarray netCDF4
  

• 预加载数据到内存：先将Dask数组加载到内存验证数据完整性，再导出：

  ds_emean.load()  # 若此处报错，说明数据本身有问题
  ds_emean.to_netcdf(path=file_out, mode='w', format='NETCDF4')
  

内容的提问来源于stack exchange，提问作者Julienne Stroeve