使用WebClient或HttpClient将文件分片上传到Amazon S3的实现问题

首先定义分片上传结果存储类，用于第四步完成上传时传入S3要求的分片编号+ETag对应列表：

public class PartUploadResult
{
    // S3要求分片编号从1开始计数
    public int PartNumber { get; set; }
    // S3返回的分片ETag值
    public string ETag { get; set; }
}

然后定义分片大小常量，和你要求的5MB一致：

private const int PART_SIZE = 5 * 1024 * 1024;

单分片上传方法实现（基于你现有HttpWebRequest写法适配） #

public PartUploadResult UploadSinglePart(string preSignedUrl, string filePath, long partStartPosition, int partSize, int partNumber)
{
    HttpWebRequest httpRequest = WebRequest.Create(preSignedUrl) as HttpWebRequest;
    httpRequest.Method = "PUT";
    // 标记当前分片的内容长度，避免超出分片大小
    httpRequest.ContentLength = partSize;

    using (Stream dataStream = httpRequest.GetRequestStream())
    {
        byte[] buffer = new byte[8192];
        int totalBytesRead = 0;
        using FileStream fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read);
        // 跳转到当前分片的起始读取位置
        fileStream.Seek(partStartPosition, SeekOrigin.Begin);
        
        while (totalBytesRead < partSize)
        {
            int bytesToRead = Math.Min(buffer.Length, partSize - totalBytesRead);
            int bytesRead = fileStream.Read(buffer, 0, bytesToRead);
            if (bytesRead == 0) break;
            dataStream.Write(buffer, 0, bytesRead);
            totalBytesRead += bytesRead;
        }
    }

    using HttpWebResponse response = httpRequest.GetResponse() as HttpWebResponse;
    // S3返回的ETag默认带双引号，需要去除
    string etag = response.Headers["ETag"].Trim('"');
    return new PartUploadResult
    {
        PartNumber = partNumber,
        ETag = etag
    };
}

批量分片上传调用逻辑 #

你已经生成了各分片对应的预签名URL列表，直接遍历调用即可：

public List<PartUploadResult> UploadAllParts(string filePath, List<string> preSignedUrls)
{
    List<PartUploadResult> uploadResults = new List<PartUploadResult>();
    FileInfo fileInfo = new FileInfo(filePath);
    long fileLength = fileInfo.Length;

    for (int i = 0; i < preSignedUrls.Count; i++)
    {
        // 分片编号从1开始，符合S3要求
        int partNumber = i + 1;
        long partStart = i * PART_SIZE;
        int currentPartSize = (int)Math.Min(PART_SIZE, fileLength - partStart);
        string preSignedUrl = preSignedUrls[i];
        
        var partResult = UploadSinglePart(preSignedUrl, filePath, partStart, currentPartSize, partNumber);
        uploadResults.Add(partResult);
    }

    return uploadResults;
}

拿到uploadResults列表后，直接传给你已经实现的第四步完成分片上传的方法即可。

可选优化 #

如果你使用的是.NET Core/.NET 5及以上版本，推荐用HttpClient代替HttpWebRequest，性能和可维护性更好，单分片上传参考写法如下：

// 建议全局声明一个HttpClient实例，不要每次请求都新建
private static readonly HttpClient _httpClient = new HttpClient();

public async Task<PartUploadResult> UploadSinglePartAsync(string preSignedUrl, string filePath, long partStartPosition, int partSize, int partNumber)
{
    using FileStream fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read);
    fileStream.Seek(partStartPosition, SeekOrigin.Begin);
    // 构造分片内容流
    using var partContent = new StreamContent(fileStream, partSize);
    var response = await _httpClient.PutAsync(preSignedUrl, partContent);
    response.EnsureSuccessStatusCode();
    string etag = response.Headers.ETag.Tag.Trim('"');
    return new PartUploadResult
    {
        PartNumber = partNumber,
        ETag = etag
    };
}

注意事项 #

• S3分片编号必须从1开始计数，不要用0作为第一个分片的编号
• 单个分片最小不能小于5MB（最后一个分片除外），否则完成上传时会报错
• 如果需要支持断点续传，可将已上传成功的分片结果持久化存储，下次上传时跳过已完成的分片即可

内容的提问来源于stack exchange，提问作者JMR