如何确定IEnumerable字节大小以设置合理CSV批量上传大小？

问题背景 #

需要将记录序列化为CSV后通过有文件大小限制的外部API上传，当前基于CsvHelper实现了内存写入逻辑，批量大小固定为50000条，但由于泛型对象<T>的字段内容长度差异，经常出现单批文件超出API限制导致上传失败的问题。

现有内存写入实现：

using CsvHelper;

public async Task<byte[]> WriteToMemoryAsync<T>(IEnumerable<T> recordsToWrite) where T : class
{
    using (var memoryStream = new MemoryStream())
    {
        using (var writer = new StreamWriter(memoryStream))
        using (var csv = new CsvWriter(writer, new CultureInfo("sv-SE")))
        {
            await csv.WriteRecordsAsync(recordsToWrite);
        }
        return memoryStream.ToArray();
    }
}

现有批量处理实现：

public async Task<Dictionary<int, byte[]>> BatchWriteToMemoryAsync<T>(IEnumerable<T> recordsToWrite) where T : class
{
    var maxBatchSize = 50_000;

    var nrOfBatches = (int)Math.Ceiling((double)recordsToWrite.Count() / maxBatchSize);

    Dictionary<int, byte[]> records = new();
    for (int batchNr = 0; batchNr < nrOfBatches; batchNr++)
    {
        records.Add(batchNr, await WriteToMemoryAsync<T>(recordsToWrite));
    }

    return records;
}

疑问 #

• 是否可以获取IEnumerable<T>的字节大小以估算文件大小，从而确定合理的批量数？
• 是否有其他可行的实现方案？

编辑补充 #

已尝试动态试探批量大小的方案，但遇到问题：检查流长度前已刷新写入记录，导致流大小超出限制。测试代码如下，失败时首个批量大小达1009B：

[Fact]
public void WhenWritingToMemoryInBatches_ShouldEnsureFileSizeDoesNotExceedLimit()
{
    //Arrange
    var records = GenerateTestRecords(100);

    var fileSizeLimit = 1_000;  //1000B limit

    var csvHandler = new CsvHandler();

    //Act
    var file = csvHandler.BatchWriteToMemory(records, fileSizeLimit);

    //Assert
    Assert.All(file, f => Assert.True(f.Length < fileSizeLimit, $"Expected fileSize to be less than {fileSizeLimit}. Actual fileSize was {f.Length}."));
}

private IEnumerable<TestRecord> GenerateTestRecords(int amountOfRecords)
{
    List<TestRecord> records = new();
    for (int i = 0; i < amountOfRecords; i++)
    {
        records.Add(new TestRecord
        {
            StringType = $"String {i}",
            IntType = 1,
        });
    }
    return records;
}

private class TestRecord
{
    public string? StringType { get; set; }
    public int IntType { get; set; }
}

解决方案 #

关于直接获取IEnumerable<T>字节大小的可行性 #

无法直接通过IEnumerable<T>的内存大小估算CSV文件大小：对象在内存中的存储大小和它序列化为CSV后的字节数没有直接对应关系，CSV大小取决于字段内容长度、分隔符、换行符、文化格式（如日期、数字的本地化）等因素。

可行实现方案 #

方案1：动态试探+回滚（精准控制单批大小）

核心逻辑是逐条添加记录，每次添加后试探当前批次的CSV字节大小，一旦接近限制就回滚最后一条记录，保存当前批次后开始新批次。解决测试中溢出问题的关键是flush后检查大小，溢出则回滚。

实现代码：

public class CsvHandler
{
    private readonly CultureInfo _csvCulture = new CultureInfo("sv-SE");

    public List<byte[]> BatchWriteToMemory<T>(IEnumerable<T> records, int fileSizeLimit) where T : class
    {
        var batches = new List<byte[]>();
        var currentBatch = new List<T>();

        foreach (var record in records)
        {
            currentBatch.Add(record);
            // 生成当前批次的CSV并检查大小
            using var tempStream = new MemoryStream();
            using var writer = new StreamWriter(tempStream, Encoding.UTF8);
            using var csv = new CsvWriter(writer, _csvCulture);
            
            csv.WriteRecords(currentBatch);
            writer.Flush(); // 必须flush才能获取准确的字节数

            if (tempStream.Length >= fileSizeLimit)
            {
                // 移除导致溢出的最后一条记录，保存当前有效批次
                currentBatch.RemoveAt(currentBatch.Count - 1);
                batches.Add(ConvertBatchToBytes(currentBatch));
                currentBatch.Clear();
                // 将溢出的记录加入新批次
                currentBatch.Add(record);
            }
        }

        // 处理剩余的最后一批记录
        if (currentBatch.Any())
        {
            batches.Add(ConvertBatchToBytes(currentBatch));
        }

        return batches;
    }

    private byte[] ConvertBatchToBytes<T>(List<T> batch) where T : class
    {
        using var memoryStream = new MemoryStream();
        using var writer = new StreamWriter(memoryStream, Encoding.UTF8);
        using var csv = new CsvWriter(writer, _csvCulture);
        
        csv.WriteRecords(batch);
        writer.Flush();
        return memoryStream.ToArray();
    }
}

方案2：样本估算平均大小（高效但有误差）

适合记录格式相对统一的场景：先取少量样本记录生成CSV，计算单条记录的平均字节大小，再结合文件限制估算批量数，同时预留冗余空间避免溢出。

实现代码：

public async Task<Dictionary<int, byte[]>> BatchWriteByEstimatedSizeAsync<T>(IEnumerable<T> records, int fileSizeLimit) where T : class
{
    var recordList = records.ToList();
    if (!recordList.Any())
        return new Dictionary<int, byte[]>();

    // 取样本计算单条记录平均大小
    var sampleCount = Math.Min(100, recordList.Count);
    var sampleBatch = recordList.Take(sampleCount).ToList();
    var sampleBytes = await WriteToMemoryAsync(sampleBatch);
    var avgBytesPerRecord = (double)sampleBytes.Length / sampleCount;

    // 预留15%冗余空间，避免实际大小超出限制
    var safeLimit = fileSizeLimit * 0.85;
    var maxBatchSize = (int)Math.Floor(safeLimit / avgBytesPerRecord);
    maxBatchSize = Math.Max(1, maxBatchSize); // 确保至少包含1条记录

    var batches = new Dictionary<int, byte[]>();
    for (var i = 0; i < recordList.Count; i += maxBatchSize)
    {
        var batch = recordList.Skip(i).Take(maxBatchSize).ToList();
        batches.Add(i / maxBatchSize, await WriteToMemoryAsync(batch));
    }

    return batches;
}

测试问题修正说明 #

测试中出现的溢出问题，是因为原实现未处理"添加记录后flush导致大小超出限制"的情况。方案1通过每次添加一条记录就试探大小，一旦超出就回滚最后一条，确保每个批次的CSV字节数严格低于限制，能通过你的测试用例。

内容的提问来源于stack exchange，提问作者FinneVirta