You need to enable JavaScript to run this app. 使用HttpClient批量计算上千URL远程文件大小的进度实现及SSL异常处理
优惠活动
大模型
产品
解决方案
定价
更多

使用HttpClient批量计算上千URL远程文件大小的进度实现及SSL异常处理

批量统计远程文件大小的常见问题与解决方案

问题场景

需要批量爬取1000余个URL对应的远程文件大小做总大小统计,希望在任务运行时实时展示已处理任务数和总任务量的进度,但只要在代码中加入进度展示逻辑,就会触发SSL连接类异常。

触发的完整报错栈

"Unhandled exception. System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.IO.IOException:  Received an unexpected EOF or 0 bytes from the transport stream.
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)"

问题根因排查

为什么加进度展示逻辑就会触发SSL异常

  • C#中调用async方法时,方法体会同步执行到第一个未完成的await点才返回任务对象。原代码在任务创建循环中加入Console.Title更新逻辑时,每创建一个请求任务就会触发一次控制台同步IO操作,拉长了HTTPS请求发起的间隔,请求逐次发出,不会触发服务端的连接防护。
  • 把进度更新逻辑移到后续的任务等待循环后,任务创建循环没有任何同步阻塞操作,1000余个HTTPS请求会在极短时间内全部发起,瞬间打满目标站点的SSL握手队列与连接配额,被服务端直接断开传输流,触发Received an unexpected EOF报错。
  • 配置ServerCertificateCustomValidationCallback仅作用于证书合法性校验环节,无法解决传输层被对端主动断开的问题,因此不会消除这类异常。

进度展示的最佳实现方式

不要一次性创建并启动所有请求任务,采用有限并发+原子计数的模式,既保证进度实时准确,又避免瞬时请求过密:

  • 设置单主机并发数上限,通过信号量控制同时发起的请求数量,避免打满服务端连接配额
  • 使用Interlocked做原子计数,每当一个请求完成时递增计数器,同步更新进度展示
  • 进度更新逻辑绑定在单个请求的完成回调中,不要绑定在任务启动或外层循环的逻辑里,避免打乱请求调度节奏

优化后的实现参考

核心业务逻辑

#region GET REMOTE FILE SIZE
public static async Task<string> GetRemoteFileSize(this List<string> urls) {
    long totalSize = 0;
    int processedCount = 0;
    int totalCount = urls.Count;
    // 设置并发上限,避免瞬时请求过密
    const int maxConcurrency = 20;
    var semaphore = new SemaphoreSlim(maxConcurrency);
    var tasks = new List<Task>();

    foreach (var url in urls) {
        await semaphore.WaitAsync();
        tasks.Add(Task.Run(async () => {
            try {
                long? size = await GetTotalBytes(url);
                Interlocked.Add(ref totalSize, size.GetValueOrDefault());
                // 单个任务完成时更新进度,不阻塞调度
                int current = Interlocked.Increment(ref processedCount);
                Console.Title = $"已处理 {current} / {totalCount}";
            } finally {
                semaphore.Release();
            }
        }));
    }
    await Task.WhenAll(tasks);
    return totalSize.FormatFileSize();
}

private static async Task<long?> GetTotalBytes(string url) {
    Uri uri = new Uri(url, UriKind.Absolute);
    uri.SetServiceManagerConnection();
    using var request = new HttpRequestMessage(HttpMethod.Head, uri);
    request.AddAllCommonHeaders(uri);
    using var response = await Client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false);
    if (!response.IsSuccessStatusCode) {
        throw new InvalidDataException($"请求异常: 状态码 {(int)response.StatusCode}");
    }
    return response.Content.Headers.ContentLength.GetValueOrDefault();
}
#endregion

工具类代码

static string FormatBytes(this long bytes) {
        var unit = 1024;
        if (bytes < unit) { return $"{bytes} B"; }
        var exp = (int)(Math.Log(bytes) / Math.Log(unit));
        return $"{bytes / Math.Pow(unit, exp):F2} {("KMGTPE")[exp - 1]}B";
    }
    public static string FormatFileSize(this long bytes) => bytes.FormatBytes();

 public static void SetServiceManagerConnection(this Uri uri) {
        var sp = ServicePointManager.FindServicePoint(uri);
        sp.ConnectionLimit = 20; // 单个端点的连接数上限,默认值为2
        sp.UseNagleAlgorithm = false; // Nagle算法通过减少网络中传输的数据包数量提升TCP/IP网络效率
        sp.Expect100Continue = false; // 在发送大体积请求前提前校验远端端点可用状态,节省带宽
        sp.ConnectionLeaseTimeout = (int)TimeSpan.FromMinutes(1).TotalMilliseconds; // 连接1分钟超时
    }

HttpClient初始化优化

.NET Core/.NET 5+版本中,ServicePointManager的配置对HttpClient的全局影响有限,推荐直接使用SocketsHttpHandler配置连接池参数、超时时间、SSL协议,替代旧的HttpClientHandler:

static readonly HttpClient Client = new(new SocketsHttpHandler {
    Proxy = null,
    UseProxy = false,
    SslOptions = new System.Net.Security.SslOptions {
        EnabledSslProtocols = System.Security.Authentication.SslProtocols.Tls12 | System.Security.Authentication.SslProtocols.Tls13,
        RemoteCertificateValidationCallback = (sender, cert, chain, errors) => true
    },
    PooledConnectionLimit = 20,
    PooledConnectionLifetime = TimeSpan.FromMinutes(1),
    UseNagling = false,
    AutomaticDecompression = System.Net.DecompressionMethods.All
});

其他注意事项

  • 不要把进度更新逻辑放在任务启动的循环里,避免任务调度节奏被UI渲染操作打断
  • 单端点并发数建议不超过30,避免被目标站点的WAF拦截
  • EOF类SSL报错本质是传输层被对端主动断开,不是证书校验问题,单纯信任所有证书无法解决这类异常

内容的提问来源于stack exchange,提问作者mark12345

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.11.09 22:54:09