在R中处理大文本文件：提速分句及关键词筛选的方法

Hey there, let's break down how to fix the crashes and speed up your workflow without upgrading hardware. The main issues here are memory overload from loading all data at once and inefficient non-vectorized operations that drag down performance. Here's a step-by-step optimized approach:

1. Ditch Heavy Corpus Packages for Lightweight File Reading #

The tm VCorpus adds unnecessary metadata overhead that wastes memory and time. Instead, read files directly with readr (faster than base R) and process them one at a time to avoid loading everything into RAM.

2. Use Vectorized String Operations (Avoid separate_rows for Large Data) #

tidyr::separate_rows can be slow on massive text chunks. Instead, use stringr::str_split to split sentences, then unnest them efficiently. We'll also switch to stringr::str_detect for faster keyword matching than grep.

3. Efficiently Flag Matching Sentences + Context #

Instead of generating index lists with grep and adjusting for neighbors, use vectorized lag() and lead() to flag rows that need to be kept (matching keyword, or adjacent to a match). This avoids costly index manipulation.

4. Process Files in Batches (Write to Disk Immediately) #

Never keep all 160 processed files in memory. Process one file, filter it, write the result to disk, then discard it from memory and move to the next.

Full Optimized Code #

library(readr)
library(stringr)
library(dplyr)
library(purrr)

# Clean up keywords (removed extra spaces to avoid missed matches)
sckeywords <- c("Affordable Housing", "Benefit The Masses",
                "Charitability", "Charitable", "Charitably", "Charities", "Charity")
pat <- paste0(sckeywords, collapse = '|')

# Function to process a single file
process_single_file <- function(file_path) {
  # Read entire file as a single string (faster for large text)
  raw_text <- read_file(file_path)
  
  # Split into sentences (handles . followed by space or newline)
  sentences <- str_split(raw_text, "\\.\\s+")[[1]]
  
  # Convert to dataframe with doc_id
  df <- tibble(
    doc_id = basename(file_path),
    text = sentences
  )
  
  # Flag rows to keep: match keyword, or previous/next row matches
  df <- df %>%
    mutate(
      has_keyword = str_detect(text, regex(pat, ignore_case = TRUE)),
      keep = has_keyword | lag(has_keyword, default = FALSE) | lead(has_keyword, default = FALSE)
    ) %>%
    filter(keep) %>%
    select(-has_keyword, -keep) # Clean up helper columns
  
  return(df)
}

# Get list of all text files in your directory
file_list <- list.files("test", pattern = "\\.txt$", full.names = TRUE)

# Process files one by one, append results to a single CSV
walk(file_list, function(file) {
  processed_df <- process_single_file(file)
  write_csv(processed_df, "filtered_sentences.csv", append = TRUE)
  # Force garbage collection to free up memory immediately
  gc()
})

Additional Performance Boosts #

• Switch to data.table: For even faster processing and lower memory usage, replace dplyr with data.table—it’s optimized for large datasets. Here’s the filtering step adapted for data.table:

  library(data.table)
  dt <- as.data.table(df)
  dt[, has_keyword := str_detect(text, regex(pat, ignore_case = TRUE))]
  dt[, keep := has_keyword | shift(has_keyword, type = "lag", fill = FALSE) | shift(has_keyword, type = "lead", fill = FALSE)]
  dt <- dt[keep == TRUE, .(doc_id, text)]
  

• Use Parquet Instead of CSV: For smaller file sizes and faster read/write speeds, use arrow::write_parquet instead of write_csv—it’s far more efficient for tabular data.
• Trim Text Early: Add text = str_trim(text) to the dataframe creation step to remove unnecessary whitespace, reducing memory usage slightly.

Why This Works #

• Memory Efficiency: Processing one file at a time means we never load all 160 files into RAM. The gc() call forces R to free up memory after each file, preventing leaks.
• Vectorized Operations: stringr and dplyr/data.table operations are vectorized, which is exponentially faster than looping or non-vectorized index manipulation.
• Reduced Overhead: Skipping tm VCorpus removes bloated metadata that was slowing down your initial dataframe conversion.

With these changes, you should avoid crashes and see a significant speedup—processing each file should take far less than 10 minutes, and you won’t max out your 16GB RAM.

内容的提问来源于stack exchange，提问作者Alex