REDCapR包redcap_read()读取5000+条记录时丢失观测及字段异常问题
REDCapR读取大数据库时丢失记录并出现未知NA列问题
我有一个包含5000+条记录的REDCap数据库,尝试通过REDCap API和R语言的REDCapR包获取每条记录的部分变量。原本使用redcap_read()就能轻松实现,但现在返回如下错误:
── Column specification ─────────────────────────────────────────────────────── cols( record_id = col_double(), redcap_repeat_instrument = col_logical(), redcap_repeat_instance = col_logical(), first_name = col_character(), last_name = col_character(), `NA` = col_logical() ) Error in redcap_read(redcap_uri = retrieve_credential_local(my_file_path, : There are 100 subject(s) that are missing rows in the returned dataset. REDCap's PHP code is likely trying to process too much text in one bite. Common solutions this problem are: - specifying only the records you need (w/ `records`) - specifying only the fields you need (w/ `fields`) - specifying only the forms you need (w/ `forms`) - specifying a subset w/ `filter_logic` - reduce `batch_size
默认批量大小(batch_size)为100,控制台显示丢失的记录数刚好等于这个参数值。
以下代码曾在200-300条记录的小型REDCap项目中正常运行:
library(REDCapR) df <- redcap_read( redcap_uri = my_redcap_uri, token = my_token, fields = c( "record_id", "first_name", "last_name"), raw_or_label_headers = 'raw', verbose = TRUE )$data
我设置了verbose=TRUE排查问题,发现每个批次都返回HTTP状态码200。根据redcap_read()的文档和源码,所有批次数据会通过dplyr::bind_rows()合并,之后R会检查行是否丢失,触发上述错误。
另外还有两个奇怪的现象:
- 在REDCap的API Playground中执行完全相同的请求,没有丢失任何行,所有指定的
record_id、first_name和last_name都能正常返回 - 仅请求
record_id时执行成功,返回的列只有record_id、redcap_repeat_instrument和redcap_repeat_instance;但同时请求first_name和last_name时,会出现一个未在fields参数中指定的logical类型NA列,同时触发丢失记录的错误。仅请求record_id的代码和返回结果如下:
df <- redcap_read( redcap_uri = my_redcap_uri, token = my_token, fields = c("record_id"), raw_or_label_headers = 'raw', verbose = TRUE )$data head(df)
── Column specification ─────────────────────────────────────────────────────── cols( record_id = col_double(), redcap_repeat_instrument = col_logical(), redcap_repeat_instance = col_logical() ) record_id redcap_repeat_instrument redcap_repeat_instance 1 NA NA 2 NA NA 3 NA NA 4 NA NA 5 NA NA 6 NA NA 7 NA NA 8 NA NA 9 NA NA 10 NA NA
请问这可能是什么原因导致的?有没有解决办法?
内容的提问来源于stack exchange,提问作者cristian-vargas
相关产品推荐
相关产品推荐

