PowerShell流式处理大文件：首行数据移至末尾方案问询

Basic Streaming Approach (Simple CSVs) #

If your CSV doesn't contain fields with embedded newlines, you can process the file line-by-line without loading the entire content into memory:

import java.io.*;

public class CsvRowMover {
    public static void moveFirstDataRowToEnd(String inputPath, String outputPath) throws IOException {
        try (BufferedReader reader = new BufferedReader(new FileReader(inputPath));
             BufferedWriter writer = new BufferedWriter(new FileWriter(outputPath))) {

            // Read and write header (skip this block if your CSV has no header)
            String header = reader.readLine();
            if (header != null) {
                writer.write(header);
                writer.newLine();
            }

            // Store the first data row (single line, minimal memory usage)
            String firstDataRow = reader.readLine();
            if (firstDataRow == null) return; // Exit if file has no data rows

            // Stream remaining rows directly to output
            String line;
            while ((line = reader.readLine()) != null) {
                writer.write(line);
                writer.newLine();
            }

            // Append the stored first data row at the end
            writer.write(firstDataRow);
        }
    }
}

Robust Streaming (Handles Multi-Line Fields) #

For CSVs with quoted fields containing newlines, use a dedicated CSV parser that supports streaming to avoid breaking rows incorrectly. Example with OpenCSV:

import com.opencsv.CSVReader;
import com.opencsv.CSVWriter;
import java.io.*;

public class RobustCsvRowMover {
    public static void moveFirstDataRowToEnd(String inputPath, String outputPath) throws IOException {
        try (CSVReader reader = new CSVReader(new FileReader(inputPath));
             CSVWriter writer = new CSVWriter(new FileWriter(outputPath))) {

            // Read and write header (skip if no header)
            String[] header = reader.readNext();
            if (header != null) writer.writeNext(header);

            // Store first data row
            String[] firstDataRow = reader.readNext();
            if (firstDataRow == null) return;

            // Stream remaining rows to output
            String[] nextRow;
            while ((nextRow = reader.readNext()) != null) {
                writer.writeNext(nextRow);
            }

            // Append first data row
            writer.writeNext(firstDataRow);
        }
    }
}

• StringBuilder: A mutable in-memory character buffer optimized for string manipulation. It’s not tied to I/O—just a tool for building strings efficiently without creating intermediate String objects.
• StringWriter: A Writer implementation that wraps a StringBuilder internally. It’s designed to work with APIs that expect a stream-based output (like logging or serialization), but it still stores all data in RAM.
• Critical similarity: Both are in-memory structures, so using them for huge files will still cause memory overflow—this is why your Method 2 failed.

• Avoid full-file loading: Always process data in chunks or line-by-line using buffered streams to minimize heap usage.
• Use temporary files: If you need to hold intermediate data, write it to a temp file instead of in-memory buffers like StringBuilder or StringWriter.
• Leverage streaming parsers: For structured formats like CSV, use parsers that read one row at a time instead of loading all rows into collections.
• Clean up resources: Use try-with-resources (as in code examples) to ensure streams are closed promptly, freeing up file handles and memory.
• Monitor heap usage: Tools like JConsole or VisualVM can help identify memory bottlenecks or leaks during processing.

内容的提问来源于stack exchange，提问作者immobile2