使用arrow与dplyr处理超大型Parquet数据集遇内存错误求解决方案

数据集概况 #

• 超大规模数据集：包含10亿+行、41个变量，拆分为3040个Parquet文件，总大小440GB
• individual_positions为个人-职位级别数据，每个唯一user_id可对应多个职位，需求是统计每个用户的职位数n_positions，并获取该字段的分布（即拥有1个、2个、3个……职位的用户数量）

尝试的处理代码 #

使用R语言的arrow包和dplyr进行大数据处理，代码如下：

individual_positions <- arrow::open_dataset(file_path)

individual_positions %>% 
  group_by(user_id) %>% # 约5~9亿个唯一用户
  summarize(n_positions = n()) %>% 
  count(n_positions, sort = TRUE) %>% 
  collect()

报错信息 #

运行上述代码后触发内存相关错误，具体报错如下：

Error in `compute.arrow_dplyr_query()`:
! Invalid: Negative buffer resize: -2147483584
Backtrace:
 1. ... %>% collect()
 3. arrow:::collect.arrow_dplyr_query(.)
 4. arrow:::compute.arrow_dplyr_query(x)

会话信息 #

─ Session info ──────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.3.2 (2023-10-31 ucrt)
 os       Windows 11 x64 (build 22621)
 system   x86_64, mingw32
 ui       RStudio
 language (EN)
 collate  Chinese (Simplified)_China.utf8
 ctype    Chinese (Simplified)_China.utf8
 tz       Asia/Shanghai
 date     2024-02-03
 rstudio  2023-12-1+402 Ocean Storm (desktop)
 pandoc   NA

─ Packages ──────────────────────────────────────────────────────────────────────────────────────────────────────
 package     * version  date (UTC) lib source
 arrow       * 14.0.0.2 2023-12-02 [1] CRAN (R 4.3.2)
 assertthat    0.2.1    2019-03-21 [1] CRAN (R 4.3.2)
 bit           4.0.5    2022-11-15 [1] CRAN (R 4.3.2)
 bit64         4.0.5    2020-08-30 [1] CRAN (R 4.3.2)
 cli           3.6.2    2023-12-11 [1] CRAN (R 4.3.2)
 clipr         0.8.0    2022-02-22 [1] CRAN (R 4.3.2)
 colorspace    2.1-0    2023-01-23 [1] CRAN (R 4.3.2)
 dplyr       * 1.1.4    2023-11-17 [1] CRAN (R 4.3.2)
 fansi         1.0.6    2023-12-08 [1] CRAN (R 4.3.2)
 forcats     * 1.0.0    2023-01-29 [1] CRAN (R 4.3.2)
 generics      0.1.3    2022-07-05 [1] CRAN (R 4.3.2)
 ggplot2     * 3.4.4    2023-10-12 [1] CRAN (R 4.3.2)
 glue          1.7.0    2024-01-09 [1] CRAN (R 4.3.2)
 gtable        0.3.4    2023-08-21 [1] CRAN (R 4.3.2)
 hms           1.1.3    2023-03-21 [1] CRAN (R 4.3.2)
 knitr         1.45     2023-10-30 [1] CRAN (R 4.3.2)
 lifecycle     1.0.4    2023-11-07 [1] CRAN (R 4.3.2)
 lubridate   * 1.9.3    2023-09-27 [1] CRAN (R 4.3.2)
 magrittr      2.0.3    2022-03-30 [1] CRAN (R 4.3.2)
 munsell       0.5.0    2018-06-12 [1] CRAN (R 4.3.2)
 pillar        1.9.0    2023-03-22 [1] CRAN (R 4.3.2)
 pkgconfig     2.0.3    2019-09-22 [1] CRAN (R 4.3.2)
 purrr       * 1.0.2    2023-08-10 [1] CRAN (R 4.3.2)
 R6            2.5.1    2021-08-19 [1] CRAN (R 4.3.2)
 readr       * 2.1.5    2024-01-10 [1] CRAN (R 4.3.2)
 rlang         1.1.3    2024-01-10 [1] CRAN (R 4.3.2)
 rstudioapi    0.15.0   2023-07-07 [1] CRAN (R 4.3.2)
 scales        1.3.0    2023-11-28 [1] CRAN (R 4.3.2)
 sessioninfo   1.2.2    2021-12-06 [1] CRAN (R 4.3.2)
 stringi       1.8.3    2023-12-11 [1] CRAN (R 4.3.2)
 stringr     * 1.5.1    2023-11-14 [1] CRAN (R 4.3.2)
 tibble      * 3.2.1    2023-03-20 [1] CRAN (R 4.3.2)
 tidyr       * 1.3.1    2024-01-24 [1] CRAN (R 4.3.2)
 tidyselect    1.2.0    2022-10-10 [1] CRAN (R 4.3.2)
 tidyverse   * 2.0.0    2023-02-22 [1] CRAN (R 4.3.2)
 timechange    0.3.0    2024-01-18 [1] CRAN (R 4.3.2)
 tzdb          0.4.0    2023-05-12 [1] CRAN (R 4.3.2)
 utf8          1.2.4    2023-10-22 [1] CRAN (R 4.3.2)
 vctrs         0.6.5    2023-12-01 [1] CRAN (R 4.3.2)
 withr         3.0.0    2024-01-16 [1] CRAN (R 4.3.2)
 xfun          0.41     2023-11-01 [1] CRAN (R 4.3.2)

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────

问题 #

1. 请解释"Negative buffer resize: -2147483584"错误的具体含义
2. 请提供可行的解决方案，包括其他语言或工具包的替代方案

内容的提问来源于stack exchange，提问作者Yi Chen