在R的foreach循环中用dbFetch执行多查询时出现‘外部指针无效’错误,求原因
问题描述
我正在使用R的foreach包加速通过SQL和dbFetch语法从服务器下载数据的循环,代码如下:
#library packages library(RPostgres) library(dplyr) library(foreach) library(doParallel) #Setup backend to use many processors totalCores = detectCores() #Leave one core to avoid overload your computer cluster <- makeCluster(totalCores[1]-1) registerDoParallel(cluster) # connect the server wrds <- dbConnect(Postgres(),host='wrds-pgdata.wharton.upenn.edu', port=9737,user=userid,password=pw,dbname='wrds',sslmode='require') # Download Data from server foreach (j = 1:length(Datetemp), .packages = "RPostgres") %dopar% { date <- as.character(Datetemp[j]) query <- paste0("select * from ", "taqmsec.ctm_", date, " where sym_root = '", sym, "' and ", "time_m between '09:30:00.000' and '16:00:00.000'") res <- dbSendQuery(wrds, query) data <- dbFetch(res, n = -1) dbClearResult(res) write.csv(data, row.names = FALSE, file = paste0(sympath, "/", sym, "_", date, ".csv")) print(paste0("Finished Stock ", sym, " for Date ", date)) }
运行错误
> task 1 failed - "external pointer is not valid"
错误原因
- 数据库连接无法跨集群节点共享:主进程中创建的
wrds数据库连接是外部指针对象,这类对象无法被序列化传递给doParallel的集群子进程。子进程拿到的是无效的指针引用,调用dbSendQuery时就会触发该错误。 - 集群子进程的独立性:
makeCluster创建的每个子进程都是独立的R会话,不会继承主进程的数据库连接资源,必须在每个子进程内部单独建立连接才能正常操作数据库。
内容的提问来源于stack exchange,提问作者Clay ZHAI
相关产品推荐
相关产品推荐

