C#分割大分隔文件时最后一行触发OutOfMemoryException问题

问题描述 #

我需要编写C#函数分割大分隔文件，解决ETL流程中2.7GB（含1238951行）文件的处理瓶颈：每个小文件限制最大行数，若原文件带表头，则每个新文件都要写入表头。

尝试了两种被推荐为内存高效的方案：

• 使用StreamReader+StreamWriter逐行读写
• 使用File.ReadLines+File.WriteAllLines流式处理

两种方案在小文件测试中正常，但处理目标大文件时，均在处理到1238950行（仅剩最后一行）时触发OutOfMemoryException，而电脑配备32GB内存，并未耗尽。

代码与报错信息 #

方案1：StreamReader/StreamWriter实现 #

internal void SplitFileWithStream(string localFilePath, bool hasHeader)
{
    try
    {
        var targetDirectory = Path.GetDirectoryName(localFilePath);
        var fileNameWithoutExtension = Path.GetFileNameWithoutExtension(localFilePath);
        var extension = Path.GetExtension(localFilePath);
        var fileSuffix = 0;
        var maxLinesPerFile = 100000;
        string? header = null;
        using (var sr = new StreamReader(localFilePath))
        {
            while (!sr.EndOfStream)
            {
                var lineNumber = 0;
                var newFileName = $"{fileNameWithoutExtension}__split_{(++fileSuffix)}{extension}";
                var newFilePath = Path.Combine(targetDirectory, newFileName);
                if (File.Exists(newFilePath))
                    File.Delete(newFilePath);
                using (var sw = new StreamWriter(newFilePath))
                {
                    if (!sr.EndOfStream && hasHeader)
                    {
                        if (header == null)
                        {
                            header = sr.ReadLine();
                            maxLinesPerFile++; //add one to max for header
                        }
                        sw.WriteLine(header);
                        lineNumber++;
                    }
                    while (!sr.EndOfStream && lineNumber < maxLinesPerFile)
                    {
                        sw.WriteLine(sr.ReadLine());
                        lineNumber++;
                    }
                    sw.Close();
                }
            }
            sr.Close();
        }
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.ToString());
    }
}

报错信息：

System.OutOfMemoryException: Exception of type
'System.OutOfMemoryException' was thrown.    at
System.Text.StringBuilder.ToString()    at
System.IO.StreamReader.ReadLine()    at
MyFileUtilities.FileSplitter.SplitFileWithStream(String localFilePath, Boolean hasHeader)
in C:\Repos\MySolution\MyFileUtilitites\FileSplitter.cs:line 341

方案2：File.ReadLines/File.WriteAllLines实现 #

internal void SplitFileWithReadLines(string localFilePath, bool hasHeader)
{
    try
    {
        var targetDirectory = Path.GetDirectoryName(localFilePath);
        var fileNameWithoutExtension = Path.GetFileNameWithoutExtension(localFilePath);
        var fileName = Path.GetFileName(localFilePath);
        var extension = Path.GetExtension(localFilePath);
        var fileSuffix = 0;
        var maxLinesPerFile = 100000;
        long position = 0;
        string? header = null;
        int i = 0;
        int skip = 0;
        string workingFilePath = localFilePath;
        while (true)
        {
            int take = maxLinesPerFile;
            if (hasHeader && header == null)
            {
                header = File.ReadLines(localFilePath).Take(1).First();
                skip = 1;
            }
            var linesToSplice = File.ReadLines(workingFilePath).Skip(skip).Take(take);
            if (!linesToSplice.Any())
                return;
            linesToSplice.Append(Environment.NewLine); //added just to see if it would try to write the actual last line, but did not work.
            var newFileName = $"{fileNameWithoutExtension}__split_{(++i)}{extension}";
            var newFilePath = Path.Combine(targetDirectory, newFileName);
            File.WriteAllLines(newFilePath, linesToSplice);
            skip += take;
        }
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.ToString());
    }
}

报错信息：

System.OutOfMemoryException: Exception of type
'System.OutOfMemoryException' was thrown.    at
System.Text.StringBuilder.ToString()    at
System.IO.StreamReader.ReadLine()    at
System.IO.ReadLinesIterator.MoveNext()    at
System.Linq.Enumerable.EnumerablePartition`1.MoveNext()    at
System.IO.File.InternalWriteAllLines(TextWriter writer, IEnumerable`1
contents)    at System.IO.File.WriteAllLines(String path,
IEnumerable`1 contents)    at
MyFileUtilities.FileSplitter.SplitFileWithReadLines(String
localFilePath, Boolean hasHeader) in
C:\Repos\MySolution\MyFileUtilitites\FileSplitter.cs\FileSplitter.cs:line
402

额外尝试 #

将File.ReadLines(workingFilePath).Skip(skip).Take(take)转为ToList()，仍触发异常：

System.OutOfMemoryException: Exception of type
'System.OutOfMemoryException' was thrown.    at
System.Text.StringBuilder.ToString()    at
System.IO.StreamReader.ReadLine()    at
System.IO.ReadLinesIterator.MoveNext()    at
System.Linq.Enumerable.EnumerablePartition`1.ToList()    at
System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)    at
MyFileUtilities.FileSplitter.SplitFileWithReadLines(String
localFilePath, Boolean hasHeader) in ...

注：已知Linux命令等工具可实现，但需要C#解决方案集成到自有工具库。

问题分析与解决办法 #

核心原因 #

报错栈指向StreamReader.ReadLine()中的StringBuilder.ToString()，说明某一行的长度异常大，导致StreamReader为读取该行分配的StringBuilder内存超过了单个对象的限制——即使总内存充足，连续内存块不足或单个字符串长度超限都会触发异常。

你的情况是最后一行报错，大概率是原文件最后一行缺失换行符，StreamReader试图把文件剩余的所有内容当作一行读取，直接撑爆内存。

解决办法 #

1. 修复原文件的换行问题

检查目标大文件的最后一行，确认是否缺失换行符，手动添加后再处理。

2. 限制每行的最大读取长度

改用固定缓冲区读取，避免一次性读取超大行，示例代码如下：

internal void SplitFileWithBufferedRead(string localFilePath, bool hasHeader)
{
    const int bufferSize = 4096; // 4KB缓冲区，可根据实际调整
    var maxLinesPerFile = 100000;
    var targetDirectory = Path.GetDirectoryName(localFilePath);
    var fileNameWithoutExtension = Path.GetFileNameWithoutExtension(localFilePath);
    var extension = Path.GetExtension(localFilePath);
    var fileSuffix = 0;
    string? header = null;
    int currentLineCount = 0;
    StreamWriter? currentWriter = null;

    try
    {
        using var sr = new StreamReader(localFilePath, bufferSize: bufferSize);
        char[] buffer = new char[bufferSize];
        StringBuilder lineBuilder = new StringBuilder();
        int charsRead;

        while ((charsRead = sr.Read(buffer, 0, buffer.Length)) > 0)
        {
            for (int i = 0; i < charsRead; i++)
            {
                char c = buffer[i];
                if (c == '\n' || c == '\r')
                {
                    // 处理一行结束
                    if (lineBuilder.Length > 0 || c == '\n')
                    {
                        ProcessLine(lineBuilder.ToString(), ref currentWriter, ref currentLineCount, 
                            ref fileSuffix, targetDirectory, fileNameWithoutExtension, extension, 
                            maxLinesPerFile, hasHeader, ref header);
                        lineBuilder.Clear();
                    }
                    // 跳过\r后面的\n，避免重复处理
                    if (c == '\r' && i + 1 < charsRead && buffer[i + 1] == '\n')
                    {
                        i++;
                    }
                }
                else
                {
                    lineBuilder.Append(c);
                    // 限制单行长，防止内存溢出
                    if (lineBuilder.Length > 10_000_000) // 限制1000万字符，可调整
                    {
                        ProcessLine(lineBuilder.ToString(), ref currentWriter, ref currentLineCount, 
                            ref fileSuffix, targetDirectory, fileNameWithoutExtension, extension, 
                            maxLinesPerFile, hasHeader, ref header);
                        lineBuilder.Clear();
                    }
                }
            }
        }

        // 处理文件末尾剩余的未换行内容
        if (lineBuilder.Length > 0)
        {
            ProcessLine(lineBuilder.ToString(), ref currentWriter, ref currentLineCount, 
                ref fileSuffix, targetDirectory, fileNameWithoutExtension, extension, 
                maxLinesPerFile, hasHeader, ref header);
        }

        currentWriter?.Close();
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.ToString());
        currentWriter?.Close();
    }
}

private void ProcessLine(string line, ref StreamWriter? currentWriter, ref int currentLineCount,
    ref int fileSuffix, string targetDirectory, string fileNameWithoutExtension, string extension,
    int maxLinesPerFile, bool hasHeader, ref string? header)
{
    // 初始化或切换文件
    if (currentWriter == null || currentLineCount >= maxLinesPerFile)
    {
        currentWriter?.Close();
        fileSuffix++;
        var newFilePath = Path.Combine(targetDirectory, $"{fileNameWithoutExtension}__split_{fileSuffix}{extension}");
        if (File.Exists(newFilePath)) File.Delete(newFilePath);
        currentWriter = new StreamWriter(newFilePath);
        currentLineCount = 0;

        // 写入表头
        if (hasHeader)
        {
            if (header == null)
            {
                header = line;
                currentWriter.WriteLine(header);
                currentLineCount++;
                return; // 表头行不占用数据行配额
            }
            else
            {
                currentWriter.WriteLine(header);
                currentLineCount++;
            }
        }
    }

    // 写入数据行
    currentWriter.WriteLine(line);
    currentLineCount++;
}

3. 确保项目以64位运行

32位进程最多只能使用约2GB内存，更容易触发溢出。在Visual Studio中修改项目设置：

• 右键项目 -> 属性 -> 生成 -> 平台目标 -> 选择x64

4. 优化方案2的逻辑（不推荐）

方案2中File.ReadLines().Skip(skip).Take(take)每次循环都会从头读取文件并跳过大量行，随着skip值增大，内存开销线性增长。若要保留该方式，应记录文件字节偏移量，直接定位到目标位置读取，避免重复扫描文件。

内容的提问来源于stack exchange，提问作者zBomb