使用HttpClient批量计算上千URL远程文件大小的进度实现及SSL异常处理
批量统计远程文件大小的常见问题与解决方案
问题场景
需要批量爬取1000余个URL对应的远程文件大小做总大小统计,希望在任务运行时实时展示已处理任务数和总任务量的进度,但只要在代码中加入进度展示逻辑,就会触发SSL连接类异常。
触发的完整报错栈
"Unhandled exception. System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Received an unexpected EOF or 0 bytes from the transport stream. at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)"
问题根因排查
为什么加进度展示逻辑就会触发SSL异常
- C#中调用async方法时,方法体会同步执行到第一个未完成的await点才返回任务对象。原代码在任务创建循环中加入
Console.Title更新逻辑时,每创建一个请求任务就会触发一次控制台同步IO操作,拉长了HTTPS请求发起的间隔,请求逐次发出,不会触发服务端的连接防护。 - 把进度更新逻辑移到后续的任务等待循环后,任务创建循环没有任何同步阻塞操作,1000余个HTTPS请求会在极短时间内全部发起,瞬间打满目标站点的SSL握手队列与连接配额,被服务端直接断开传输流,触发
Received an unexpected EOF报错。 - 配置
ServerCertificateCustomValidationCallback仅作用于证书合法性校验环节,无法解决传输层被对端主动断开的问题,因此不会消除这类异常。
进度展示的最佳实现方式
不要一次性创建并启动所有请求任务,采用有限并发+原子计数的模式,既保证进度实时准确,又避免瞬时请求过密:
- 设置单主机并发数上限,通过信号量控制同时发起的请求数量,避免打满服务端连接配额
- 使用
Interlocked做原子计数,每当一个请求完成时递增计数器,同步更新进度展示 - 进度更新逻辑绑定在单个请求的完成回调中,不要绑定在任务启动或外层循环的逻辑里,避免打乱请求调度节奏
优化后的实现参考
核心业务逻辑
#region GET REMOTE FILE SIZE public static async Task<string> GetRemoteFileSize(this List<string> urls) { long totalSize = 0; int processedCount = 0; int totalCount = urls.Count; // 设置并发上限,避免瞬时请求过密 const int maxConcurrency = 20; var semaphore = new SemaphoreSlim(maxConcurrency); var tasks = new List<Task>(); foreach (var url in urls) { await semaphore.WaitAsync(); tasks.Add(Task.Run(async () => { try { long? size = await GetTotalBytes(url); Interlocked.Add(ref totalSize, size.GetValueOrDefault()); // 单个任务完成时更新进度,不阻塞调度 int current = Interlocked.Increment(ref processedCount); Console.Title = $"已处理 {current} / {totalCount}"; } finally { semaphore.Release(); } })); } await Task.WhenAll(tasks); return totalSize.FormatFileSize(); } private static async Task<long?> GetTotalBytes(string url) { Uri uri = new Uri(url, UriKind.Absolute); uri.SetServiceManagerConnection(); using var request = new HttpRequestMessage(HttpMethod.Head, uri); request.AddAllCommonHeaders(uri); using var response = await Client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false); if (!response.IsSuccessStatusCode) { throw new InvalidDataException($"请求异常: 状态码 {(int)response.StatusCode}"); } return response.Content.Headers.ContentLength.GetValueOrDefault(); } #endregion
工具类代码
static string FormatBytes(this long bytes) { var unit = 1024; if (bytes < unit) { return $"{bytes} B"; } var exp = (int)(Math.Log(bytes) / Math.Log(unit)); return $"{bytes / Math.Pow(unit, exp):F2} {("KMGTPE")[exp - 1]}B"; } public static string FormatFileSize(this long bytes) => bytes.FormatBytes(); public static void SetServiceManagerConnection(this Uri uri) { var sp = ServicePointManager.FindServicePoint(uri); sp.ConnectionLimit = 20; // 单个端点的连接数上限,默认值为2 sp.UseNagleAlgorithm = false; // Nagle算法通过减少网络中传输的数据包数量提升TCP/IP网络效率 sp.Expect100Continue = false; // 在发送大体积请求前提前校验远端端点可用状态,节省带宽 sp.ConnectionLeaseTimeout = (int)TimeSpan.FromMinutes(1).TotalMilliseconds; // 连接1分钟超时 }
HttpClient初始化优化
.NET Core/.NET 5+版本中,ServicePointManager的配置对HttpClient的全局影响有限,推荐直接使用SocketsHttpHandler配置连接池参数、超时时间、SSL协议,替代旧的HttpClientHandler:
static readonly HttpClient Client = new(new SocketsHttpHandler { Proxy = null, UseProxy = false, SslOptions = new System.Net.Security.SslOptions { EnabledSslProtocols = System.Security.Authentication.SslProtocols.Tls12 | System.Security.Authentication.SslProtocols.Tls13, RemoteCertificateValidationCallback = (sender, cert, chain, errors) => true }, PooledConnectionLimit = 20, PooledConnectionLifetime = TimeSpan.FromMinutes(1), UseNagling = false, AutomaticDecompression = System.Net.DecompressionMethods.All });
其他注意事项
- 不要把进度更新逻辑放在任务启动的循环里,避免任务调度节奏被UI渲染操作打断
- 单端点并发数建议不超过30,避免被目标站点的WAF拦截
- EOF类SSL报错本质是传输层被对端主动断开,不是证书校验问题,单纯信任所有证书无法解决这类异常
内容的提问来源于stack exchange,提问作者mark12345
相关产品推荐
相关产品推荐

