为何sapply(df, sum)性能优于base::colSums？附基准测试

我发现使用sapply(df, sum)的速度比base::colSums(df)更快，甚至对矩阵使用base::colSums(M)时也是如此。这是为什么？我是否忽略了什么？一直都是这样吗？

基准测试 #

在10K×10K数据框上的基准测试结果如下：

$ Rscript --vanilla speed_test.R 
Unit: milliseconds
      expr       min        lq      mean   median       uq       max neval  cld
    sapply 125.76717 125.77749 126.21036 125.7878 126.4320 127.07610     3 a   
   colSums 288.09562 293.57566 298.28873 299.0557 303.3853 307.71486     3  b  
 colSums_M 137.68780 139.08794 141.25548 140.4881 143.0393 145.59055     3   c 
  colSums2  55.49845  55.86153  56.04262  56.2246  56.3147  56.40479     3    d

  # A tibble: 4 × 13
  expression      min   median `itr/sec` mem_alloc `gc/sec` n_itr  n_gc total_time result memory               time           gc              
  <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl> <int> <dbl>   <bch:tm> <list> <list>               <list>         <list>          
1 sapply      124.8ms  124.9ms      8.01   440.8KB     0        5     0      625ms <NULL> <Rprofmem [6 × 3]>   <bench_tm [5]> <tibble [5 × 3]>
2 colSums     285.7ms  329.5ms      3.04   763.4MB     3.04     2     2      659ms <NULL> <Rprofmem [403 × 3]> <bench_tm [2]> <tibble [2 × 3]>
3 colSums_M     131ms    131ms      7.63    78.2KB     0        4     0      524ms <NULL> <Rprofmem [1 × 3]>   <bench_tm [4]> <tibble [4 × 3]>
4 colSums2     55.4ms   55.8ms     17.9     78.2KB     0        9     0      502ms <NULL> <Rprofmem [1 × 3]>   <bench_tm [9]> <tibble [9 × 3]>

测试环境：R版本4.4.2（2024-10-31），AlmaLinux 9.5系统，NETLIB或OPENBLAS-OPENMP（无影响），AMD Ryzen 7 7700X处理器。

补充测试 #

在旧款AMD FX(tm)-8350处理器上进行相同基准测试，结果如下：

$ Rscript --vanilla speed_test.R 
## col sums:
Unit: milliseconds
      expr      min       lq     mean   median       uq      max neval  cld
    sapply 169.5607 169.7373 169.8201 169.9138 169.9499 169.9859     3 a   
   colSums 573.5435 575.7917 576.6825 578.0399 578.2520 578.4640     3  b  
 colSums_M 130.5275 130.6009 130.6255 130.6744 130.6745 130.6746     3   c 
  colSums2 148.7892 149.0359 149.4866 149.2826 149.8354 150.3881     3    d

# A tibble: 4 × 13
  expression      min   median `itr/sec` mem_alloc `gc/sec` n_itr  n_gc total_time result memory               time           gc              
  <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl> <int> <dbl>   <bch:tm> <list> <list>               <list>         <list>          
1 sapply        170ms    174ms      5.79   440.8KB     0        3     0      518ms <NULL> <Rprofmem [6 × 3]>   <bench_tm [3]> <tibble [3 × 3]>
2 colSums       703ms    703ms      1.42   763.4MB     1.42     1     1      703ms <NULL> <Rprofmem [399 × 3]> <bench_tm [1]> <tibble [1 × 3]>
3 colSums_M     131ms    131ms      7.63    78.2KB     0        4     0      524ms <NULL> <Rprofmem [1 × 3]>   <bench_tm [4]> <tibble [4 × 3]>
4 colSums2      150ms    150ms      6.65    78.2KB     0        4     0      602ms <NULL> <Rprofmem [1 × 3]>   <bench_tm [4]> <tibble [4 × 3]>

或许base::colSums尚未针对新硬件优化？

测试代码 #

set.seed(42)

m <- 1e4; n <- 1e4
M <- matrix(rnorm(m*n), m, n)
df <- data.frame(M)

options(width=200)

microbenchmark::microbenchmark(
  sapply=sapply(df, sum),
  colSums=colSums(df),
  colSums_M=colSums(M),  ## <-- USING MATRIX INPUT TO AVOID as.matrix() OVERHEAD
  colSums2=matrixStats::colSums2(M),
  times=3L,
  check='equivalent'
  ) |> print()

bench::mark(sapply=sapply(df, sum),
            colSums=colSums(df),
            colSums_M=colSums(M),
            colSums2=matrixStats::colSums2(M), check=FALSE)

内容的提问来源于stack exchange，提问作者jay.sf