在R中如何将计算结果直接流式传入fread，无需写入临时CSV文件？

Great question! In R, you absolutely can skip writing intermediate CSV files like temp.csv by using in-memory text connections—this works very similarly to Python's io.stream by passing data directly between functions without touching the filesystem. Here are a few straightforward ways to do this:

方法1：使用textConnection配合写入函数 #

You can create an in-memory text connection, write your processed data to it, then read directly from that connection with fread (or your custom wrapper function).

library(data.table)
library(dplyr)

# 1. 读取原始输入数据
input_data <- fread("input.csv")

# 2. 执行计算：为所有数值列加10
processed_data <- input_data %>% 
  mutate(across(where(is.numeric), ~ .x + 10))

# 3. 创建内存文本连接并写入处理后的数据
con <- textConnection("processed_csv_text", "w")
write.csv(processed_data, con, row.names = FALSE)  # 写入到内存连接
close(con)  # 关闭连接完成写入

# 4. 直接从内存文本读取到fread
streamed_data <- fread(processed_csv_text)

# 如果你有自定义的CSV读取函数，直接传入内存文本即可
custom_csv_reader <- function(csv_content) {
  # 这里是你的自定义处理逻辑，比如指定列类型、编码等
  fread(csv_content, colClasses = c("numeric", "character"), encoding = "UTF-8")
}

final_processed_data <- custom_csv_reader(processed_csv_text)

方法2：更简洁的管道式捕获输出 #

Use capture.output() to directly capture the CSV output as a character string, then pass it to fread:

# 捕获write.csv的输出为字符串
processed_csv_text <- capture.output(write.csv(processed_data, row.names = FALSE))

# 把字符串拼接成完整的CSV文本并读取
streamed_data <- fread(paste(processed_csv_text, collapse = "\n"))

方法3：用data.table::fwrite优化性能 #

If you're working with larger datasets, fwrite (from data.table) is faster than base R's write.csv. You can pair it with a text connection too:

con <- textConnection(NULL, "w")
fwrite(processed_data, con, row.names = FALSE)
processed_csv_text <- textConnectionValue(con)
close(con)

streamed_data <- fread(processed_csv_text)

注意事项 #

• These methods keep data in memory, so for extremely large datasets, you might run into memory constraints—but for most typical use cases, this is far more efficient than writing/reading from disk.
• Make sure your custom reading function accepts a character string (the CSV content) as input, since that's what we're passing instead of a file path.

内容的提问来源于stack exchange，提问作者AndreyIto