.NET Core 6下如何跳过CSV首行后追加写入Azure Blob

场景与问题 #

在.NET Core 6环境下，调用多个API端点下载CSV文件并追加写入Azure Blob容器。每份CSV都包含相同的表头行，且表头行末尾必有换行符。追加后Blob文件中会出现多份重复表头，同时文件可能达数GB级，无法全部加载到内存处理。

原简化代码：

using Stream blobStream = await blockBlobClient.OpenWriteAsync(true);
{
    for (int i = 0; i < 3; i++)
    {
        using HttpResponseMessage response = await client.GetAsync(downloadUrls[i], HttpCompletionOption.ResponseHeadersRead);

        Stream sourceStream = response.Content.ReadAsStream();
        sourceStream.CopyTo(blobStream);
    }
}

解决方案 #

核心思路：仅保留第一份CSV的表头，后续所有CSV跳过首行后再写入Blob，全程采用流式处理，避免加载整个文件到内存。

方法一：基于字符流处理（编码已知时使用）

适合明确CSV编码（如UTF8）的场景，代码简洁易读：

using Stream blobStream = await blockBlobClient.OpenWriteAsync(true);
{
    for (int i = 0; i < downloadUrls.Length; i++)
    {
        using HttpResponseMessage response = await client.GetAsync(downloadUrls[i], HttpCompletionOption.ResponseHeadersRead);
        response.EnsureSuccessStatusCode(); // 确保API请求成功

        using Stream sourceStream = await response.Content.ReadAsStreamAsync();
        using StreamReader reader = new StreamReader(sourceStream);

        // 第一份文件保留表头，后续文件跳过表头行
        if (i > 0)
        {
            await reader.ReadLineAsync();
        }

        // 批量读取剩余内容并写入Blob，平衡内存占用与性能
        char[] buffer = new char[8192];
        int charsRead;
        while ((charsRead = await reader.ReadAsync(buffer, 0, buffer.Length)) > 0)
        {
            byte[] bytes = System.Text.Encoding.UTF8.GetBytes(buffer, 0, charsRead);
            await blobStream.WriteAsync(bytes, 0, bytes.Length);
        }
    }
}

方法二：基于字节流处理（兼容多编码场景）

直接操作字节流，不依赖字符编码，避免编码不一致导致的解析问题：

using Stream blobStream = await blockBlobClient.OpenWriteAsync(true);
{
    for (int i = 0; i < downloadUrls.Length; i++)
    {
        using HttpResponseMessage response = await client.GetAsync(downloadUrls[i], HttpCompletionOption.ResponseHeadersRead);
        response.EnsureSuccessStatusCode();

        using Stream sourceStream = await response.Content.ReadAsStreamAsync();

        if (i > 0)
        {
            // 逐字节读取直到找到换行符（兼容\r\n和\n格式）
            int currentByte;
            bool hasSkippedHeader = false;
            while ((currentByte = await sourceStream.ReadByteAsync()) != -1)
            {
                if (currentByte == '\n')
                {
                    hasSkippedHeader = true;
                    break;
                }
                // 处理Windows换行符\r\n
                if (currentByte == '\r')
                {
                    int nextByte = await sourceStream.ReadByteAsync();
                    if (nextByte == '\n')
                    {
                        hasSkippedHeader = true;
                        break;
                    }
                    else
                    {
                        throw new InvalidDataException("CSV表头行末尾未包含有效换行符");
                    }
                }
            }

            if (!hasSkippedHeader)
            {
                throw new InvalidDataException("未找到CSV表头行的换行符");
            }
        }

        // 将剩余流直接复制到Blob
        await sourceStream.CopyToAsync(blobStream);
    }
}

关键说明 #

• 两种方法均采用流式处理，内存占用仅为缓冲区大小，适合GB级大文件
• HttpCompletionOption.ResponseHeadersRead确保仅读取响应头后就开始处理流，无需等待整个文件下载完成
• response.EnsureSuccessStatusCode()避免因API请求失败导致无效流写入

内容的提问来源于stack exchange，提问作者chris