求助：使用R将超10000个不同列名CSV文件合并为单一Data frame

Hey there! Since you're new to R and tackling the task of merging 10,000+ CSV files with different column names into a single data frame, let's walk through a straightforward, efficient solution that's perfect for your scenario.

Step 1: Set Up Required Packages #

First, we'll use the tidyverse collection of packages—it makes reading and combining data way easier. If you haven't installed it yet, run this:

install.packages("tidyverse")
library(tidyverse)

Step 2: Get Paths to All CSV Files #

Start by pointing R to your target folder. Replace "your_folder_path" with the actual path to your folder (e.g., "C:/MyCSVs" on Windows or "/home/user/MyCSVs" on macOS/Linux). We'll grab all CSV file paths at once:

# Get full paths to all CSV files; add recursive=TRUE if you have CSVs in subfolders
csv_file_paths <- list.files(
  path = "your_folder_path",
  pattern = "\\.csv$",  # Only match files ending with .csv
  full.names = TRUE     # Return full file paths instead of just filenames
)

Step 3: Batch Read and Merge Files #

The key here is using map_dfr() from the purrr package (part of tidyverse) to read each CSV and automatically combine them into a single data frame. bind_rows() (which map_dfr() uses under the hood) will match columns by name and fill missing values with NA for columns that don't exist in a particular CSV—exactly what you need for mixed column names.

Run this to create your master dataset:

# Read all CSVs and combine into one data frame
master_dataset <- map_dfr(csv_file_paths, ~read_csv(.x))

Handling Edge Cases #

If you run into issues like encoding problems (e.g., non-English characters showing up as gibberish) or odd delimiters, tweak the read_csv() call:

# Example: For CSV files with GBK encoding
master_dataset <- map_dfr(csv_file_paths, ~read_csv(.x, locale = locale(encoding = "GBK")))

# Example: For CSV files using semicolons instead of commas
master_dataset <- map_dfr(csv_file_paths, ~read_csv(.x, delim = ";"))

Step 4: Verify the Merged Dataset #

Once the merge is done, take a quick look to make sure everything worked:

# View basic info about the dataset
glimpse(master_dataset)

# List all column names
colnames(master_dataset)

# Check how many rows and columns you have
dim(master_dataset)

Tips for Working with 10,000+ Files #

• Save Memory: If your combined dataset is huge, use the vroom package instead of read_csv—it's faster and uses less memory:

  install.packages("vroom")
  library(vroom)
  master_dataset <- map_dfr(csv_file_paths, ~vroom(.x))
  

• Catch Errors: If some files fail to read (e.g., corrupted files), use safely() to skip them without breaking the whole process:

  # Create a safe version of read_csv that won't crash on errors
  safe_read_csv <- safely(read_csv)
  
  # Read all files, capturing successes and errors
  read_results <- map(csv_file_paths, safe_read_csv)
  
  # Extract only the successfully read data frames
  successful_dfs <- read_results %>% 
    map("result") %>% 
    discard(is.null)
  
  # Combine the successful ones
  master_dataset <- bind_rows(successful_dfs)
  
  # Check which files failed (if any)
  failed_files <- read_results %>% 
    map("error") %>% 
    keep(!is.null)
  

• Track Progress: For large batches, add a progress bar to see how things are going:

  install.packages("progressr")
  library(progressr)
  handlers(global = TRUE)  # Turn on progress bars globally
  
  with_progress({
    progress <- progressor(along = csv_file_paths)
    master_dataset <- map_dfr(csv_file_paths, ~{
      progress()  # Update the progress bar
      read_csv(.x)
    })
  })
  

内容的提问来源于stack exchange，提问作者Rajesh Ghosal