如何将R的data.table按多个指定列拆分为data.table列表？代码问题求解

Great question! Let's break down why your current approach isn't working, then walk through the most efficient, memory-friendly solutions tailored for data.table.

Why your split_by approach fails #

The problem with split(dt, list(split_by)) is that split_by is a vector of strings (like "dt$carat"), not the actual column values from your data.table. When you wrap these strings in list(), you're passing literal text as grouping keys instead of the underlying data—R has no way to map those strings to the real columns in dt.

Step 1: Define your target columns cleanly #

First, let's simplify how we select columns to split on. setdiff is a concise base R alternative to your custom %ni% operator:

cols_to_split <- setdiff(colnames(dt), c("x", "y"))

Option 1: Use data.table's optimized split method #

data.table has its own built-in split method that’s far faster and more memory-efficient than base R’s version. It accepts a by parameter for column names, which aligns perfectly with your needs:

# Split and keep grouping columns (default behavior)
dt_to_list <- split(dt, by = cols_to_split)

# Or exclude grouping columns from the resulting subsets
dt_to_list <- split(dt, by = cols_to_split, keep.by = FALSE)

This method leverages data.table's internal grouping logic to minimize unnecessary data copies, making it ideal for large datasets.

Option 2: Ultra-efficient grouped list creation #

For even better performance (especially with massive tables), use data.table's core grouping syntax to directly generate a list of subsets. This cuts out overhead from the split wrapper:

# Generate list of subsets (excludes grouping columns by default)
dt_to_list <- dt[, .(list(.SD)), by = cols_to_split]$V1

# If you need grouping columns included in each subset:
dt_to_list <- dt[, .(list(copy(.SD))), by = cols_to_split]$V1

Here, .SD represents the subset of data for each group. Wrapping it in list() stores each subset as an element in a list column, and extracting V1 gives you the final list of data.table objects. Use copy() only if you plan to modify subsets later—omit it for minimal memory usage when just reading data.

Key advantages of these approaches #

• Speed: data.table's grouping operations are implemented in C, so they’re drastically faster than base R for large datasets.
• Memory efficiency: Both methods avoid redundant data copies, which is critical when working with big tables.
• Readability: Using column names directly makes the code easier to maintain than constructing string references.

内容的提问来源于stack exchange，提问作者J. Doe.