如何高效合并不同行数的RData列表为data.frame并去除NA行？

Hey there! Your current code gets the job done, but all those repetitive manual conversions and merges are definitely dragging down performance. Let's simplify this with more efficient data.table workflows to speed things up.

核心问题分析 #

Your original approach manually converts each list element to a data.table, sets keys one by one, and merges them sequentially. This isn't just tedious—it also misses out on data.table's built-in optimizations for batch processing and joins.

优化方案1：批量转换 + Reduce 批量合并 #

This approach eliminates redundant code by using lapply to process all list elements at once, then Reduce to merge them in a single optimized pass:

# Load the remote data (direct URL load works in R)
load("https://stepik.org/media/attachments/course/724/all_data.Rdata")
library(data.table)

# 1. Convert all list elements to data.tables and set keys in one go
dt_list <- lapply(all_data, function(x) {
  dt <- as.data.table(x)
  setkey(dt, id) # Set key once per table for lightning-fast joins
  dt
})

# 2. Merge all data.tables sequentially using Reduce
all_day <- Reduce(function(x, y) x[y], dt_list)

# 3. Remove rows with any NA values
all_day <- na.omit(all_day)

优化方案2：更简洁的合并（无需手动设置键） #

If you prefer not to set keys explicitly, you can use merge with by = "id" inside Reduce—data.table will still optimize the join under the hood:

load("https://stepik.org/media/attachments/course/724/all_data.Rdata")
library(data.table)

dt_list <- lapply(all_data, as.data.table)
all_day <- Reduce(function(x, y) merge(x, y, by = "id", all.x = TRUE), dt_list)
all_day <- na.omit(all_day)

为什么这更快？ #

• Batch processing: lapply handles all list conversions in a single vectorized operation, cutting down on redundant function calls.
• Optimized joins: Reduce streamlines the merge process into a single loop, and data.table's join logic (whether key-based or explicit by) is tuned for speed, especially with large datasets.
• Less overhead: Eliminating 7 separate setkey and merge calls removes unnecessary processing overhead.

额外小提示 #

If you're certain all your list elements have identical id values in the exact same order, you can skip merging entirely and use cbind for even faster results (only use this if you're 100% sure about consistency!):

# Only use if all id columns match perfectly in order and values
all_day <- do.call(cbind, lapply(all_data, as.data.table))
all_day <- na.omit(all_day)

内容的提问来源于stack exchange，提问作者Ekaterina