R中数据框转数组体积暴增且写入NetCDF报错的求助

我有一个存储为.RData文件的R对象，包含Lat、Lon列，以及1950年1月至2099年12月（共1800个月）每个月单像素值的列，数据框维度为810810行×1802列。我需将其转换为数组以生成NetCDF文件作为最终产物，但转换后对象体积大幅增加，且执行代码时出现错误：

Error in ncvar_put(ncnew, var, dataarray) : 
  'list' object cannot be coerced to type 'double'
Execution halted

执行输出显示：转换前数据框大小为11688853808字节，转换后数组维度为1386×585×1800，大小异常达到9466826857488224字节。此前使用相同方法未遇该问题，相关代码如下：

library(ncdf4)

# Load the RData file
load("Rcp45_bcc-csm1-1_1Month_SPEI_Df.RData")
print("RData File Loaded")

# Open the NetCDF template file to get lat and lon values
file_nc_maxT1 <- nc_open("macav2metdata_PotEvap_bcc-csm1-1_r1i1p1_rcp45_2096_2099_CONUS_monthly.nc")
nclat <- as.numeric(ncvar_get(file_nc_maxT1,"lat"))
nclon <- as.numeric(ncvar_get(file_nc_maxT1,"lon"))
nc_close(file_nc_maxT1)

# Define the filename for the new NetCDF file
filename <- "macav2metdata_1-month_SPEI_bcc-csm1-1_r1i1p1_rcp45_1950_2099_CONUS_monthly.nc"

# Create the NetCDF file
nx <- length(nclon)
ny <- length(nclat)
lon <- ncdim_def("lon", units = "degrees_east", longname = "longitude", vals = nclon)
lat <- ncdim_def("lat", units = "degrees_north", longname = "latitude", vals = nclat)
nctime <- seq(as.Date("1950-01-01"),as.Date("2099-12-31"),by="months")
time <- ncdim_def("time", units = "days since 1900-01-01 00:00:00", vals = as.numeric(nctime - as.Date("1900-01-01")))
mv <- -999 
var <- ncvar_def("SPEI", "1-month SPEI", list(lon, lat, time), longname="1-month Standardized Precipitation Evapotranspiration Index (SPEI)", mv)

ncnew <- nc_create(filename, list(var))

print(paste("The file has", ncnew$nvars,"variables"))
print(paste("The file has", ncnew$ndim,"dimensions"))

# Replace NaN and NA values with missing value (mv)
is.nan.data.frame <- function(x)
do.call(cbind, lapply(x, is.nan))

SPEIDf[is.nan(SPEIDf)] <- mv
SPEIDf[is.na(SPEIDf)] <- mv

# Convert data to array format
dataarray <- array(SPEIDf[, -c(1, 2)], dim = c(nx, ny, length(nctime)))
print("SPEIDf")
print(dim(SPEIDf))
print(object.size(SPEIDf))
print("dataarray")
print(dim(dataarray))
print(object.size(dataarray))

# Write data array to the NetCDF file
ncvar_put(ncnew, var, dataarray)

# Close the NetCDF file
nc_close(ncnew)
print("NC File Created")

核心问题分析 #

1. 数组类型错误：dataarray被创建为列表数组而非数值型数组，因为SPEIDf[, -c(1,2)]作为数据框传入array()时，会被当作列表处理，导致ncvar_put无法将其转为double类型。
2. 维度匹配错误：数据框的810810行（1386×585）对应空间像素，但直接按c(nx, ny, length(nctime))构造数组时，数据的排列顺序不匹配NetCDF的维度要求，同时未先将数据框转为矩阵（数值型），导致内存占用异常。

修正步骤 #

1. 将数据框转为数值矩阵：先提取数值列并转为矩阵，确保数据类型为数值型，避免列表数组问题。
2. 调整数组维度与排列顺序：NetCDF通常采用[lon, lat, time]或[lat, lon, time]的顺序，但数据框的行是每个像素的时间序列，需要先将矩阵转置后再重塑维度，确保空间维度与模板NC的lat/lon对应。
3. 优化缺失值处理：简化缺失值替换逻辑，避免自定义函数可能的问题。

修正后的代码 #

library(ncdf4)

# Load the RData file
load("Rcp45_bcc-csm1-1_1Month_SPEI_Df.RData")
print("RData File Loaded")

# Open the NetCDF template file to get lat and lon values
file_nc_maxT1 <- nc_open("macav2metdata_PotEvap_bcc-csm1-1_r1i1p1_rcp45_2096_2099_CONUS_monthly.nc")
nclat <- as.numeric(ncvar_get(file_nc_maxT1,"lat"))
nclon <- as.numeric(ncvar_get(file_nc_maxT1,"lon"))
nc_close(file_nc_maxT1)

# Define the filename for the new NetCDF file
filename <- "macav2metdata_1-month_SPEI_bcc-csm1-1_r1i1p1_rcp45_1950_2099_CONUS_monthly.nc"

# Create the NetCDF file
nx <- length(nclon)
ny <- length(nclat)
lon <- ncdim_def("lon", units = "degrees_east", longname = "longitude", vals = nclon)
lat <- ncdim_def("lat", units = "degrees_north", longname = "latitude", vals = nclat)
nctime <- seq(as.Date("1950-01-01"),as.Date("2099-12-31"),by="months")
time <- ncdim_def("time", units = "days since 1900-01-01 00:00:00", vals = as.numeric(nctime - as.Date("1900-01-01")))
mv <- -999 
var <- ncvar_def("SPEI", "1-month SPEI", list(lon, lat, time), longname="1-month Standardized Precipitation Evapotranspiration Index (SPEI)", mv)

ncnew <- nc_create(filename, list(var))

print(paste("The file has", ncnew$nvars,"variables"))
print(paste("The file has", ncnew$ndim,"dimensions"))

# Replace NaN and NA values with missing value (mv)
# 简化缺失值处理，直接对数值列操作
spei_values <- SPEIDf[, -c(1, 2)]
spei_values[is.na(spei_values) | is.nan(spei_values)] <- mv

# 转换为数值矩阵，转置后重塑为数组（确保时间维度在最后，空间维度匹配模板）
# 数据框每行是一个像素的所有时间点，转置后每列对应一个像素，再重塑为[lon, lat, time]
spei_matrix <- as.matrix(spei_values)
dataarray <- aperm(array(t(spei_matrix), dim = c(length(nctime), ny, nx)), c(3,2,1))

print("SPEIDf")
print(dim(SPEIDf))
print(object.size(SPEIDf))
print("dataarray")
print(dim(dataarray))
print(object.size(dataarray))

# Write data array to the NetCDF file
ncvar_put(ncnew, var, dataarray)

# Close the NetCDF file
nc_close(ncnew)
print("NC File Created")

关键说明 #

• as.matrix()：将数据框的数值列转为数值型矩阵，避免array()将数据框解析为列表。
• t(spei_matrix)：转置矩阵，让每列对应一个像素的时间序列，方便后续重塑为空间-时间维度。
• aperm()：调整数组维度顺序，确保最终数组的维度顺序与NetCDF变量定义的lon, lat, time一致。
• 修正后数组的内存占用会恢复正常（约等于原数据框大小，因为都是数值型数据），同时解决ncvar_put的类型错误。

内容的提问来源于stack exchange，提问作者Prasad Thota