计算NARR位势高度3小时方差并本地保存失败的技术求助

问题背景 #

我需要计算NOAA物理科学实验室托管的NARR逐3小时位势高度数据的方差，用于对比不同日期对应时段的位势高度时参考总体方差信息。但在将计算结果保存到本地磁盘时失败，报错NetCDF: DAP failure。

复现代码 #

import datetime as dt
import xarray as xr
import numpy as np

import warnings
warnings.filterwarnings("ignore", message = r"variable 'hgt' has multiple fill values") 

# 指定5个数据集文件（完整数据集共535个文件）
files = ['https://psl.noaa.gov/thredds/dodsC/Datasets/NARR/pressure/hgt.197901.nc',
 'https://psl.noaa.gov/thredds/dodsC/Datasets/NARR/pressure/hgt.197902.nc',
 'https://psl.noaa.gov/thredds/dodsC/Datasets/NARR/pressure/hgt.197903.nc',
 'https://psl.noaa.gov/thredds/dodsC/Datasets/NARR/pressure/hgt.197904.nc',
 'https://psl.noaa.gov/thredds/dodsC/Datasets/NARR/pressure/hgt.197905.nc']

# 使用xarray的open_mfdataset打开数据集
test = xr.open_mfdataset(files,chunks='auto')

# 选择1000hPa层并计算时间维度的标准差
# 实际场景会按每日3小时时段计算标准差，此处简化示例
test1 = test.sel(level=1000).std(dim='time')

# 先执行test.load()也无法解决问题
test1.to_netcdf('test.nc')

报错信息 #

File "/Users/gdl/MetPy/OpenDAP_test.py", line 16, in <module>                                                                 
    test.to_netcdf('test.nc')                                                                                                   
  File "/Users/gdl/opt/anaconda3/envs/MetPy_x86/lib/python3.9/site-packages/xarray/core/dataset.py", line 1957, in to_netcdf     
    return to_netcdf(  # type: ignore  # mypy cannot resolve the overloads:(                                                    
  File "/Users/gdl/opt/anaconda3/envs/MetPy_x86/lib/python3.9/site-packages/xarray/backends/api.py", line 1281, in to_netcdf     
    writes = writer.sync(compute=compute)                                                                                       
  File "/Users/gdl/opt/anaconda3/envs/MetPy_x86/lib/python3.9/site-packages/xarray/backends/common.py", line 178, in sync        
    delayed_store = chunkmanager.store(                                                                                         
  File "/Users/gdl/opt/anaconda3/envs/MetPy_x86/lib/python3.9/site-packages/xarray/core/daskmanager.py", line 211, in store      
    return store(                                                                                                               
  File "/Users/gdl/opt/anaconda3/envs/MetPy_x86/lib/python3.9/site-packages/dask/threaded.py", line 89, in get                   
    results = get_async(                                                                                                        
  File "/Users/gdl/opt/anaconda3/envs/MetPy_x86/lib/python3.9/site-packages/dask/local.py", line 511, in get_async               
    raise_exception(exc, tb)                                                                                                    
  File "/Users/gdl/opt/anaconda3/envs/MetPy_x86/lib/python3.9/site-packages/dask/local.py", line 319, in reraise                 
    raise exc                                                                                                                   
  File "/Users/gdl/opt/anaconda3/envs/MetPy_x86/lib/python3.9/site-packages/dask/local.py", line 224, in execute_task            
    result = _execute_task(task, data)                                                                                          
  File "/Users/gdl/opt/anaconda3/envs/MetPy_x86/lib/python3.9/site-packages/xarray/core/indexing.py", line 484, in __array__     
    return np.asarray(self.get_duck_array(), dtype=dtype)                                                                       
  File "/Users/gdl/opt/anaconda3/envs/MetPy_x86/lib/python3.9/site-packages/xarray/core/indexing.py", line 487, in get_duck_array                                                                                                                               
    return self.array.get_duck_array()                                                                                          
  File "/Users/gdl/opt/anaconda3/envs/MetPy_x86/lib/python3.9/site-packages/xarray/core/indexing.py", line 664, in get_duck_array                                                                                                                               
    return self.array.get_duck_array()                                                                                          
  File "/Users/gdl/opt/anaconda3/envs/MetPy_x86/lib/python3.9/site-packages/xarray/core/indexing.py", line 557, in get_duck_array                                                                                                                               
    array = array.get_duck_array()                                                                                              
  File "/Users/gdl/opt/anaconda3/envs/MetPy_x86/lib/python3.9/site-packages/xarray/coding/variables.py", line 74, in get_duck_array                                                                                                                             
    return self.func(self.array.get_duck_array())                                                                                
  File "/Users/gdl/opt/anaconda3/envs/MetPy_x86/lib/python3.9/site-packages/xarray/core/indexing.py", line 551, in get_duck_array                                                                                                                               
    array = self.array[self.key]                                                                                                
  File "/Users/gdl/opt/anaconda3/envs/MetPy_x86/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 100, in __getitem__                                                                                                                                 return indexing.explicit_indexing_adapter(
  File "/Users/gdl/opt/anaconda3/envs/MetPy_x86/lib/python3.9/site-packages/xarray/core/indexing.py", line 858, in explicit_indexing_adapter
    result = raw_indexing_method(raw_key.tuple)
  File "/Users/gdl/opt/anaconda3/envs/MetPy_x86/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 113, in _getitem
    array = getitem(original_array, key)
  File "/Users/gdl/opt/anaconda3/envs/MetPy_x86/lib/python3.9/site-packages/xarray/backends/common.py", line 73, in robust_getitem
    return array[key]
  File "src/netCDF4/_netCDF4.pyx", line 4739, in netCDF4._netCDF4.Variable.__getitem__
  File "src/netCDF4/_netCDF4.pyx", line 5688, in netCDF4._netCDF4.Variable._get
  File "src/netCDF4/_netCDF4.pyx", line 1965, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: DAP failure

解决思路 #

1. 分批次处理数据 #

不要一次性加载大量数据，按月拆分计算后再合并结果，减少单次网络请求的数据量：

import datetime as dt
import xarray as xr
import numpy as np
import warnings
warnings.filterwarnings("ignore", message=r"variable 'hgt' has multiple fill values")

files = ['https://psl.noaa.gov/thredds/dodsC/Datasets/NARR/pressure/hgt.197901.nc',
            'https://psl.noaa.gov/thredds/dodsC/Datasets/NARR/pressure/hgt.197902.nc',
            'https://psl.noaa.gov/thredds/dodsC/Datasets/NARR/pressure/hgt.197903.nc',
            'https://psl.noaa.gov/thredds/dodsC/Datasets/NARR/pressure/hgt.197904.nc',
            'https://psl.noaa.gov/thredds/dodsC/Datasets/NARR/pressure/hgt.197905.nc']

# 初始化空列表存储每月计算结果
monthly_std = []
for f in files:
    ds = xr.open_dataset(f, chunks='auto')
    std_ds = ds.sel(level=1000).std(dim='time')
    monthly_std.append(std_ds)
    ds.close()  # 关闭数据集释放资源

# 合并所有月份的结果，计算整体标准差
combined = xr.concat(monthly_std, dim='temp_dim')
final_std = combined.std(dim='temp_dim')
final_std.to_netcdf('test.nc')

2. 手动调整分块策略 #

避免自动分块导致的不合理分块，手动指定时间维度的分块大小，平衡内存与网络效率：

test = xr.open_mfdataset(files, chunks={'time': 24})  # 按24个时间步长分块

3. 先下载数据到本地再处理 #

如果网络稳定性差，直接批量下载文件到本地，再进行计算，避免长时间依赖网络连接：

# 批量下载示例（bash环境）
files=("https://psl.noaa.gov/thredds/dodsC/Datasets/NARR/pressure/hgt.197901.nc"
       "https://psl.noaa.gov/thredds/dodsC/Datasets/NARR/pressure/hgt.197902.nc"
       "https://psl.noaa.gov/thredds/dodsC/Datasets/NARR/pressure/hgt.197903.nc"
       "https://psl.noaa.gov/thredds/dodsC/Datasets/NARR/pressure/hgt.197904.nc"
       "https://psl.noaa.gov/thredds/dodsC/Datasets/NARR/pressure/hgt.197905.nc")

for file in "${files[@]}"; do
    wget "$file"
done

之后用xr.open_mfdataset加载本地文件即可。

4. 调整网络超时与缓存设置 #

延长DAP连接超时时间，增大缓存，缓解网络延迟导致的中断：

import netCDF4 as nc
import os

# 设置超时时间为3600秒
os.environ['NC_DAP_TIMEOUT'] = '3600'
# 增大缓存大小
nc.Dataset.default_chunk_cache_size = 1024*1024*100

内容的提问来源于stack exchange，提问作者gdlewen