在R中基于CSV文件创建Parquet文件目录：大文件内存外（OOM）转换方案咨询

Awesome question—dealing with massive flat files without shoving the whole thing into R's memory is exactly what the Apache Arrow ecosystem was built for, and converting CSV to Parquet is one of its most straightforward use cases. Here's a step-by-step, memory-efficient approach that keeps your data out of R's working memory entirely:

1. Install & Load the Arrow Package #

First, make sure you have the latest version of the arrow package installed (it handles all the memory-optimized under-the-hood work):

install.packages("arrow")
library(arrow)

2. Create a CSV Dataset (No Data Loaded Yet) #

Instead of reading the CSV into R with read.csv() or even read_csv_arrow(), use open_csv_dataset() to create a "virtual" pointer to your CSV file (or directory of CSVs). This doesn't load any data into memory—it just tells Arrow where to find the data and how to read it:

# Point to your single large CSV file OR a directory of multiple CSVs
csv_source <- "/path/to/your/large_data.csv" # or "/path/to/csv_directory/"

# Create the dataset object (memory footprint is tiny)
csv_dataset <- open_csv_dataset(
  csv_source,
  # Optional: Customize CSV parsing to avoid auto-inference errors
  delim = ",",
  col_types = schema(
    user_id = int64(),
    transaction_amount = float64(),
    transaction_date = date32()
  ),
  skip_rows = 1 # If you need to skip header rows or extra lines
)

3. Stream CSV to Parquet (Memory-Optimized Write) #

Use write_dataset() to convert the CSV dataset directly to Parquet. Arrow will stream the CSV in chunks, process each chunk, and write it to Parquet files—no full dataset is ever loaded into R's memory:

# Define output directory for Parquet files
parquet_output_dir <- "/path/to/output_parquet_files/"

# Write the dataset (automatically chunks data to avoid memory overload)
write_dataset(
  csv_dataset,
  path = parquet_output_dir,
  format = "parquet",
  # Optional: Control Parquet file size (adjust based on your data)
  max_rows_per_file = 1e6, # Write ~1 million rows per Parquet file
  # Optional: Partition data by a column (e.g., date) for faster future queries
  # partitioning = c("transaction_date")
)

Key Details to Note #

• Memory Efficiency: Arrow handles all processing in its own C++ engine, using only a small, fixed amount of memory for each data chunk. You won't see R's memory usage spike even with multi-GB CSV files.
• Multiple CSVs: If you have a directory of smaller CSV files, open_csv_dataset() will automatically treat them as a single dataset—no need to concatenate first.
• Parquet Benefits: Parquet is columnar, compressed, and optimized for fast analytics. Once converted, you can use open_dataset(parquet_output_dir) to query the data without loading it all into memory later.

Verify the Output #

To make sure everything worked, you can peek at the Parquet dataset without loading it:

parquet_dataset <- open_dataset(parquet_output_dir, format = "parquet")
# View first 10 rows (only loads a tiny chunk into memory)
head(parquet_dataset, n = 10)

内容的提问来源于stack exchange，提问作者Trent