使用循环构建基线数据表，是否存在更优实现方案？

Great question! Nested loops can quickly become unwieldy and slow when dealing with large datasets like yours (5000 subjects × 6 rounds × 1000 variables = 30 million data points total). Instead, we can use a tidy data workflow with tidyr and dplyr (part of the tidyverse) to reshape your data and compute summary statistics in a vectorized, efficient way.

Here's a step-by-step solution:

1. Load Required Packages #

First, install and load the tidyverse if you haven't already:

install.packages("tidyverse")
library(tidyverse)

2. Reshape Wide Data to Long Format #

Your data is in "wide" format (one column per variable-round combination). We'll convert it to "long" format, which makes grouping and summarizing trivial. We'll split column names into two new columns: variable (the core measurement name, e.g., cyl) and round (the measurement wave, e.g., r1).

# Convert wide to long format
long_df <- your_dataframe %>%
  pivot_longer(
    cols = everything(),  # Target all columns
    names_to = c("variable", "round"),
    names_pattern = "([a-zA-Z]+)(r\\d+)"  # Regex to split variable name and round
  )

Note: Adjust the names_pattern regex if your variable names include numbers (e.g., cyl1r1 would need ([a-zA-Z0-9]+)(r\\d+)).

3. Classify Variable Types #

We need to distinguish continuous, categorical, and binary variables to compute the right statistics. We can auto-detect types from your original dataframe:

# Create a lookup table for variable types
var_types <- your_dataframe %>%
  summarise(across(everything(), ~class(.x))) %>%  # Get class of each column
  pivot_longer(everything(), names_to = "colname", values_to = "type") %>%
  mutate(
    variable = str_extract(colname, "[a-zA-Z]+"),  # Extract core variable name
    type = case_when(
      type %in% c("numeric", "integer") ~ "continuous",
      type %in% c("factor", "character") ~ "categorical",
      type == "logical" ~ "binary"  # Treat logicals as binary
    )
  ) %>%
  select(variable, type) %>%
  distinct()  # Keep one entry per unique variable

# Merge type info with our long-format data
long_df <- long_df %>% left_join(var_types, by = "variable")

4. Compute Summary Statistics #

Now we can group by variable, round, and type, then compute the appropriate stats using group_modify (which lets us run different logic for each group type):

summary_stats <- long_df %>%
  group_by(variable, round, type) %>%
  group_modify(function(data, group_keys) {
    if (group_keys$type == "continuous") {
      # For continuous vars: mean, SD, and count of valid values
      data %>%
        summarise(
          mean = mean(value, na.rm = TRUE),
          sd = sd(value, na.rm = TRUE),
          n_valid = sum(!is.na(value))
        )
    } else {
      # For categorical/binary vars: frequency and percentage
      data %>%
        drop_na(value) %>%
        count(value) %>%
        mutate(
          category = value,
          percentage = (n / sum(n)) * 100,
          .keep = "unused"  # Drop original value column
        )
    }
  }) %>%
  ungroup()

Why This Is Better Than Nested Loops #

• Efficiency: Tidyverse functions are built on vectorized operations (powered by C++ under the hood), which are drastically faster than R's base loops for large datasets.
• Readability: The code clearly maps to your workflow (reshape → classify → summarize) instead of being buried in loop logic.
• Scalability: Adding new variables, rounds, or statistics only requires small tweaks, not rewriting loop structures.
• Maintainability: Debugging and modifying tidy code is far easier than untangling nested loops.

Key Notes #

• NA Handling: We use na.rm = TRUE for continuous stats and drop_na() for categorical stats—adjust this if your missing data needs special treatment.
• Binary Variables: If you want to explicitly compute "positive" rates (e.g., percentage of TRUE values), you can add a separate case for binary types in group_modify.
• Regex Precision: Double-check the names_pattern regex to ensure it correctly splits your column names. Test with str_extract(your_dataframe$colnames[1], "([a-zA-Z]+)(r\\d+)") if needed.

内容的提问来源于stack exchange，提问作者Thomas