如何优化带条件的data.table聚合函数,提升百万级数据处理速度?
问题
我有一个聚合函数,用于对分组数据求和后,根据一组条件生成标记并分配给对应分组。当前问题在于需聚合的分组数量极多,但每个分组的数据量极小,这导致即便处理中等规模数据集,聚合所需时间也过长,而该函数需适配百万行级别的数据集。我已创建如下可复现示例,其逻辑与我的聚合函数类似(我的函数包含更多条件,但性质相近):
# install.packages("palmerpenguins") library(data.table) library(palmerpenguins) # 创建数据 GROUPS <- 1:100 penguin_list <- lapply(GROUPS, \(x) data.table(group = x, penguins)) penguin_table <- rbindlist(penguin_list) # 聚合函数 aggregatePenguinMass <- function(mass, sex, ratio = 2/3){ data <- data.table(mass, sex) total <- sum(data[,mass], na.rm = TRUE) n_sex <- data[,.N, by = sex] n_male <- n_sex[sex == "male", N] n_female <- n_sex[sex == "female", N] if(n_female >= ratio * (n_male + n_female)){ return(data.table(total = total, flag = "F")) } else { return(data.table(total = total, flag = "M")) } } # 执行聚合并计时 system.time( penguin_table[, aggregatePenguinMass(body_mass_g, sex), by = .(group, species, year)] )
# user system elapsed # 2.66 0.47 7.30
如何修改该函数或聚合方式,使其速度提升一个数量级?
基准测试
Unit: milliseconds expr min lq mean median uq max neval base() 1793.4883 1803.1108 2491.32386 1821.8588 1928.4540 8195.9058 10 RBarradas1() 1554.5428 1577.5579 1614.60087 1583.7645 1637.0681 1764.4564 10 RBarradas2() 119.5481 127.6831 186.11583 131.7930 141.9157 657.2762 10 NGraham() 23.2437 23.8428 25.50738 24.5770 27.4004 30.6569 10 Miff() 427.2106 440.3240 462.32146 455.1878 462.6939 577.3803 10
内容的提问来源于Stack Exchange,提问作者Sebastian-C
相关产品推荐
相关产品推荐

