使用HttpClient批量下载列表文件失败，仅生成小文件问题排查

问题原因分析 #

1. 请求头设置错误：代码错误地将User-Agent放入Authorization请求头，不符合HTTP规范，服务器无法识别合法请求，返回几KB的错误HTML页面而非目标视频文件。
2. 进度计算逻辑颠倒：GetProgressPercentage方法参数顺序错误，导致进度值计算异常，虽不直接引发下载失败，但会破坏进度显示逻辑。
3. 未校验响应状态：下载过程未检查HTTP响应码，即使服务器返回4xx/5xx错误，仍会将错误内容写入文件，形成小文件。
4. 批量请求触发限流：短时间内发起大量请求，触发服务器反爬/限流机制，返回错误内容而非目标文件。

解决办法 #

1. 修正User-Agent请求头设置 #

替换错误的Authorization头配置，用正确方式设置User-Agent：

// 替换原Authorization头代码
client.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0 Chrome");

2. 修复进度百分比计算逻辑 #

调整参数顺序，确保进度值计算正确：

float GetProgressPercentage(long currentBytes, long totalBytes) => ((float)currentBytes / totalBytes) * 100f;

3. 添加响应状态校验 #

获取响应后立即校验状态码，确保请求成功：

using (var response = await client.GetAsync(requestUrl, HttpCompletionOption.ResponseHeadersRead))
{
    // 响应失败直接抛出异常
    response.EnsureSuccessStatusCode();
    var contentLength = response.Content.Headers.ContentLength;
    // 后续下载逻辑...
}

4. 批量下载添加请求延迟 #

循环中加入短暂延迟，避免请求过于频繁：

for (int i = 0; i < videosLinks.Count; i++)
{
    var filePath = @"d:\Test\" + i + ".mp4";
    try
    {
        using (var file = new FileStream(filePath, FileMode.Create, FileAccess.Write, FileShare.None))
            await client.DownloadDataAsync(videosLinks[i], file, progress);
        // 每次下载后延迟1秒，可按需调整
        await Task.Delay(1000);
    }
    catch (Exception ex)
    {
        Console.WriteLine($"下载第{i}个文件失败：{ex.Message}");
    }
}

5. 校验视频链接有效性 #

单独测试videosLinks中的每个URL，确保能在浏览器中正常下载完整视频。

修正后的完整代码示例 #

主方法修正后： #

async Task DownloadFile()
{
    var client = new HttpClient();
    // 正确设置User-Agent
    client.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0 Chrome");

    var progress = new Progress<float>();
    progress.ProgressChanged += Progress_ProgressChanged;

    for (int i = 0; i < videosLinks.Count; i++)
    {
        var filePath = @"d:\Test\" + i + ".mp4";
        try
        {
            using (var file = new FileStream(filePath, FileMode.Create, FileAccess.Write, FileShare.None))
                await client.DownloadDataAsync(videosLinks[i], file, progress);
            // 添加请求延迟
            await Task.Delay(1000);
        }
        catch (Exception ex)
        {
            Console.WriteLine($"下载第{i}个文件失败：{ex.Message}");
        }
    }
}

void Progress_ProgressChanged(object sender, float progress)
{
    // 确保进度值在0-100区间内
    progressBar1.Value = Math.Clamp((int)progress, 0, 100);
}

private async void button1_Click(object sender, EventArgs e)
{
    await DownloadFile();
}

扩展类修正后： #

using System;
using System.IO;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;

namespace HttpClientProgress
{
    public static class HttpClientProgressExtensions
    {
        public static async Task DownloadDataAsync(this HttpClient client, string requestUrl, Stream destination, IProgress<float> progress = null, CancellationToken cancellationToken = default(CancellationToken))
        {
            using (var response = await client.GetAsync(requestUrl, HttpCompletionOption.ResponseHeadersRead))
            {
                // 校验响应状态
                response.EnsureSuccessStatusCode();
                var contentLength = response.Content.Headers.ContentLength;
                using (var download = await response.Content.ReadAsStreamAsync())
                {
                    if (progress is null || !contentLength.HasValue)
                    {
                        await download.CopyToAsync(destination);
                        return;
                    }
                    var progressWrapper = new Progress<long>(totalBytes => progress.Report(GetProgressPercentage(totalBytes, contentLength.Value)));
                    await download.CopyToAsync(destination, 81920, progressWrapper, cancellationToken);
                }
            }

            // 修正进度计算参数顺序
            float GetProgressPercentage(long currentBytes, long totalBytes) => ((float)currentBytes / totalBytes) * 100f;
        }

        static async Task CopyToAsync(this Stream source, Stream destination, int bufferSize, IProgress<long> progress = null, CancellationToken cancellationToken = default(CancellationToken))
        {
            if (bufferSize < 0)
                throw new ArgumentOutOfRangeException(nameof(bufferSize));
            if (source is null)
                throw new ArgumentNullException(nameof(source));
            if (!source.CanRead)
                throw new InvalidOperationException($"'{nameof(source)}' is not readable.");
            if (destination == null)
                throw new ArgumentNullException(nameof(destination));
            if (!destination.CanWrite)
                throw new InvalidOperationException($"'{nameof(destination)}' is not writable.");

            var buffer = new byte[bufferSize];
            long totalBytesRead = 0;
            int bytesRead;
            while ((bytesRead = await source.ReadAsync(buffer, 0, buffer.Length, cancellationToken).ConfigureAwait(false)) != 0)
            {
                await destination.WriteAsync(buffer, 0, bytesRead, cancellationToken).ConfigureAwait(false);
                totalBytesRead += bytesRead;
                progress?.Report(totalBytesRead);
            }
        }
    }
}

内容的提问来源于stack exchange，提问作者Daniel Lip