使用Azure.Storage.Blobs .NET从Azure存储下载文件上传OneDrive遇内存问题

我目前使用Azure.Storage.Blobs v12.14.1实现从Azure Blob存储下载文件，再分块上传至OneDrive。

问题回顾 #

1. 最初代码：文件小于10MB分块大小时正常，大文件上传后OneDrive中找不到文件，无报错

var blobClient = new BlobClient({connection_string}, {container}, {file_path});
var blobSize = blobClient.GetProperties().Value.ContentLength;
var client = _clientFactory.CreateClient();

long blockSize = (10 * 1024 * 1024);
blockSize = Math.Min(blobSize, blockSize);

long currentPointer = 0;
long bytesRemaining = blobSize;
do
{
    var bytesToFetch = Math.Min(blockSize, bytesRemaining);
    await using (var blobStream = await blobClient.OpenReadAsync(currentPointer, (int)bytesToFetch, null, cancellationToken))
    {
        blobStream.Position = 0;

        var fileContent = new StreamContent(blobStream);
        fileContent.Headers.ContentLength = bytesToFetch;
        fileContent.Headers.ContentRange = new ContentRangeHeaderValue(currentPointer, currentPointer + bytesToFetch - 1, blobSize);

        var res = await client.PutAsync(fileConsentCardResponse.UploadInfo.UploadUrl, fileContent, cancellationToken);

        currentPointer += bytesToFetch;
        bytesRemaining -= bytesToFetch;
    }
}
while (bytesRemaining > 0);

2. 修改后代码：功能恢复正常，但处理1GB级大文件时内存占用过高

var blobClient = new BlobClient({connection_string}, {container}, {file_path});
var blobSize = blobClient.GetProperties().Value.ContentLength;
var client = _clientFactory.CreateClient();

long blockSize = (10 * 1024 * 1024);
blockSize = Math.Min(blobSize, blockSize);

long currentPointer = 0;
long bytesRemaining = blobSize;
do
{
    var bytesToFetch = Math.Min(blockSize, bytesRemaining);
    HttpRange range = new HttpRange(currentPointer, bytesToFetch);

    var blobStreamResult = await blobClient.DownloadStreamingAsync(
        new BlobDownloadOptions { Range = range });

    using (MemoryStream ms = new MemoryStream())
    {
        blobStreamResult.Value.Content.CopyTo(ms);
        ms.Position = 0;
        var fileContent = new StreamContent(ms);

        fileContent.Headers.ContentLength = bytesToFetch;
        fileContent.Headers.ContentRange = new ContentRangeHeaderValue(currentPointer, currentPointer + bytesToFetch - 1, blobSize);

        var res = await client.PutAsync(fileConsentCardResponse.UploadInfo.UploadUrl, fileContent, cancellationToken);
    }

    currentPointer += bytesToFetch;
    bytesRemaining -= bytesToFetch;
}
while (bytesRemaining > 0);

问题根源 #

修改后的代码将Blob下载的流完整复制到MemoryStream后再上传，每一个10MB的分块都会被加载到内存中，大文件的多个分块会导致内存占用急剧升高，甚至引发内存溢出。

优化方案：流直接传输，避免内存缓冲 #

直接使用Blob下载的原始流作为上传的数据源，不需要经过MemoryStream中转，实现流的"管道式"传输，内存中仅保留当前传输的缓冲区数据，大幅降低内存占用。

优化后的代码 #

var blobClient = new BlobClient({connection_string}, {container}, {file_path});
var blobSize = blobClient.GetProperties().Value.ContentLength;
var client = _clientFactory.CreateClient();

long blockSize = 10 * 1024 * 1024; // 10MB分块
blockSize = Math.Min(blobSize, blockSize);

long currentPointer = 0;
long bytesRemaining = blobSize;
do
{
    var bytesToFetch = Math.Min(blockSize, bytesRemaining);
    var range = new HttpRange(currentPointer, bytesToFetch);

    // 获取Blob的分块流，使用await using确保流正确释放
    await using var blobStreamResult = await blobClient.DownloadStreamingAsync(
        new BlobDownloadOptions { Range = range }, cancellationToken);
    await using var blobStream = blobStreamResult.Value.Content;

    // 直接使用Blob流创建StreamContent，无需内存中转
    var fileContent = new StreamContent(blobStream);
    fileContent.Headers.ContentLength = bytesToFetch;
    fileContent.Headers.ContentRange = new ContentRangeHeaderValue(
        currentPointer, 
        currentPointer + bytesToFetch - 1, 
        blobSize);

    // 执行上传并检查响应状态，确保分块上传成功
    var res = await client.PutAsync(
        fileConsentCardResponse.UploadInfo.UploadUrl, 
        fileContent, 
        cancellationToken);
    res.EnsureSuccessStatusCode(); // 若上传失败抛出异常，避免静默错误

    currentPointer += bytesToFetch;
    bytesRemaining -= bytesToFetch;
}
while (bytesRemaining > 0);

关键优化点 #

• 移除MemoryStream中转：直接将Blob下载流传入StreamContent，数据从Azure Blob流直接流向OneDrive的上传请求，不占用额外内存存储整个分块
• 严格管理流生命周期：使用await using确保Blob下载流在使用后及时释放，避免资源泄漏
• 增加响应检查：调用res.EnsureSuccessStatusCode()，确保分块上传过程中出现的错误能被及时捕获，避免大文件上传到一半静默失败

额外注意事项 #

• OneDrive分块上传的建议块大小为5MB-100MB，当前10MB的设置是合理的
• 确保ContentRange头的计算准确：currentPointer + bytesToFetch - 1不能超过文件总大小，最后一块需要特别注意
• 若网络环境不稳定，可添加重试逻辑，提升分块上传的可靠性

内容的提问来源于stack exchange，提问作者Ferry