替代For Loop:百万级数据动态过滤与汇总的高效方案
处理500万条数据的空间邻域高效汇总方案
我有一个包含500万条记录的数据集,结构如下:
library(dplyr) Data <- tibble( X = runif(5000000, min=200000, max=400000), Y = runif(5000000, min=400000, max=500000), UID = 1:5000000, Count = runif(5000000, min=1, max=2000) )
需要为每个UID计算四个空间范围(200m、400m、800m、1600m)内的以下指标:
- 范围内其他点的数量
- 这些点的
Count字段总和 - 密度值,计算公式:
Count总和 / (点数量 * 0.000625)
最终输出要包含每个UID及其对应四个范围的所有汇总结果。
我用for循环实现了需求,但运行耗时极长,代码如下:
Summary <- NULL for (n in 1:nrow(Data)){ Data_Row <- Data[n,] Row_200 <- Data %>% filter(between(X, Data_Row$X-200, Data_Row$X+200) & between(Y, Data_Row$Y-200, Data_Row$Y+200)) %>% summarise( Instances_200 = n(), Count_200 = sum(Count), Calculation_200 = Count_200/(Instances_200*0.000625) ) Row_400 <- Data %>% filter(between(X, Data_Row$X-400, Data_Row$X+400) & between(Y, Data_Row$Y-400, Data_Row$Y+400)) %>% summarise( Instances_400 = n(), Count_400 = sum(Count), Calculation_400 = Count_400/(Instances_400*0.000625) ) Row_800 <- Data %>% filter(between(X, Data_Row$X-800, Data_Row$X+800) & between(Y, Data_Row$Y-800, Data_Row$Y+800)) %>% summarise( Instances_800 = n(), Count_800 = sum(Count), Calculation_800 = Count_800/(Instances_800*0.000625) ) Row_1600 <- Data %>% filter(between(X, Data_Row$X-1600, Data_Row$X+1600) & between(Y, Data_Row$Y-1600, Data_Row$Y+1600)) %>% summarise( Instances_1600 = n(), Count_1600 = sum(Count), Calculation_1600 = Count_1600/(Instances_1600*0.000625) ) Summary_Row <- bind_cols(Data[n,"UID"], Row_200, Row_400, Row_800, Row_1600) Summary <- bind_rows(Summary, Summary_Row) }
我了解lapply、map这类函数,但不知道如何用它们高效完成这种动态过滤与汇总操作,求可行的解决方案。
内容的提问来源于stack exchange,提问作者Chris
相关产品推荐
相关产品推荐

