如何使用lapply函数对文件夹中所有CSV文件批量执行向后消元逐步回归

Let's break down what's causing your error and walk through a robust, scalable solution to process all your CSV files in the sample_obs folder.

Why You're Getting Error in terms(object) : object 'i' not found #

Your lapply code has two critical issues that trigger this error:

1. Incorrect model input to step(): You're passing i[["coefficients"]] instead of the full lm object. The step() function requires a complete linear model object (with its formula, data, and terms) to perform backward elimination—just passing coefficients strips away all the metadata it needs to evaluate variable removal.
2. Hardcoded scope definition: You're using a fixed all_IVs_model for the scope argument, instead of referencing the formula from the current model i in the loop. This breaks the scope that defines which variables are eligible for elimination for each individual dataset.

Fixed Batch Processing Code #

Here's a clean, scalable solution that handles any number of CSV files, fits backward elimination models, and outputs results in your requested format:

# Set your working directory to the sample_obs folder (update the path!)
setwd("path/to/your/sample_obs")

# Get a list of all CSV files in the folder
csv_files <- list.files(pattern = "\\.csv$")

# Define a function to process a single CSV file
process_single_csv <- function(file_name) {
  # Read the CSV data
  dataset <- read.csv(file_name)
  
  # Extract the dataset name (remove the .csv extension)
  ds_name <- gsub("\\.csv$", "", file_name)
  
  # Build the full model with all predictors (Y ~ all variables)
  full_lm <- lm(Y ~ ., data = dataset)
  
  # Set seed to ensure reproducible backward elimination results
  set.seed(50)
  
  # Run backward elimination
  be_model <- step(
    object = full_lm,
    direction = "backward",
    scope = formula(full_lm), # Use the current model's formula for scope
    trace = 0 # Suppress verbose step-by-step output
  )
  
  # Extract selected predictors (exclude the intercept term)
  selected_vars <- names(be_model[["coefficients"]][-1])
  
  # Format the output line as requested: "dataset-name, var1 var2 var3..."
  paste0(ds_name, ",", paste(selected_vars, collapse = " "))
}

# Batch process all CSV files using lapply
all_results <- lapply(csv_files, process_single_csv)

# Print all results to the console (one line per dataset)
cat(paste(unlist(all_results), collapse = "\n"), "\n")

# Optional: Save results to a text file for later use
writeLines(unlist(all_results), "backward_elimination_results.txt")

Key Improvements in This Code #

• Self-contained processing function: The process_single_csv function handles all steps for one file, making the code easy to debug and modify.
• Correct model handling: We pass the full full_lm object to step(), which fixes the terms(object) error.
• Dynamic scope: Uses formula(full_lm) to ensure each model's scope matches its own set of predictors.
• Automatic dataset naming: No need for a separate name list—we extract the dataset name directly from the file path.
• Scalability: Works with any number of CSV files, including your 47,000 files.

Optimizing for 47,000 Files #

Processing 47k models will take time with standard lapply. For faster execution, use parallel processing with the parallel package:

library(parallel)

# Set up a parallel cluster (use all available cores minus 1 to avoid system slowdown)
cl <- makeCluster(detectCores() - 1)

# Export the processing function to the cluster
clusterExport(cl, "process_single_csv")

# Run parallel processing
all_results_parallel <- parLapply(cl, csv_files, process_single_csv)

# Stop the cluster to free up resources
stopCluster(cl)

# Print or save results as before
cat(paste(unlist(all_results_parallel), collapse = "\n"), "\n")

内容的提问来源于stack exchange，提问作者Marlen