单例ApiAdapter并行调用时HttpClient SSL连接建立失败问题

我有一个使用HttpClient发送HTTP请求的ApiAdapter，为避免创建过多HttpClient实例，将其注册为单例模式。逐个调用请求时运行正常，但用Parallel.ForEachAsync批量创建1000个用户时，在创建30-100个后抛出如下异常：

System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.ArgumentNullException: Value cannot be null. (Parameter 'value')
   at System.Collections.CollectionBase.OnValidate(Object value)
   at System.Security.Cryptography.X509Certificates.X509CertificateCollection.OnValidate(Object value)
   at System.Collections.CollectionBase.System.Collections.IList.Add(Object value)
   at System.Security.Cryptography.X509Certificates.X509CertificateCollection.AddRange(X509CertificateCollection value)
   at System.Net.Security.CertificateHelper.GetEligibleClientCertificate(X509CertificateCollection candidateCerts)
   at System.Net.Http.HttpClientHandler.<set_ClientCertificateOptions>b__57_0(Object sender, String targetHost, X509CertificateCollection localCertificates, X509Certificate remoteCertificate, String[] acceptableIssuers)
   at System.Net.Security.SslStream.UserCertSelectionCallbackWrapper(String targetHost, X509CertificateCollection localCertificates, X509Certificate remoteCertificate, String[] acceptableIssuers)
   at System.Net.Security.SecureChannel.AcquireClientCredentials(Byte[]& thumbPrint)
   at System.Net.Security.SecureChannel.GenerateToken(ReadOnlySpan`1 inputBuffer, Byte[]& output)
   at System.Net.Security.SecureChannel.NextMessage(ReadOnlySpan`1 incomingBuffer)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at -----<Path package>----.ApiAdapter.PostRequestAsync

相关代码如下：

public async Task RequestPerson()
{
    var threadRequestPins = Enumerable.Range(0, listSize).ToList();
    await Parallel.ForEachAsync(datasToProcess, async (_, token) =>{
        var fakfurTestApiService = LoadTestKernel.Kernel.Get<IFakfurApiService>();
        var personCreated = await fakfurTestApiService.CallPostRequestAsync("<Url path>", <dataBody>);
        //CallPostRequestAsync will call ApiAdapter with PostRequestAsync() method below
    });
}

//constructor
public ApiAdapter()
{
    _cookieContainer = new CookieContainer();
    _httpClientHandler = new HttpClientHandler
    {
        CookieContainer = _cookieContainer
    };
    _httpClient = new HttpClient(_httpClientHandler);
}

private void AddCertificates()
{
    if (_certificate2Collection != null )
    {
        _httpClientHandler.ClientCertificates.AddRange(_certificate2Collection);
    }
}

public async Task<TResult> PostRequestAsync<TResult, TBodyRequest>(string testApiRelativeUrl, TBodyRequest bodyRequest,
            CancellationToken cancellationToken = default) where TResult : class
{
    var uriBuilder = BuildRequestUri(testApiRelativeUrl);
    var request = new HttpRequestMessage(HttpMethod.Post, uriBuilder.Uri);
    AddCertificate(request);
    request.Content = new StringContent(_jsonSerializer.SerializeObject(bodyRequest), Encoding.UTF8, "application/json");
    var response = await _httpClient.SendAsync(request, cancellationToken);
    return await ExtractBodyFromResponseAsync<TResult>(response);
}

问题原因 #

异常栈显示问题出在SSL证书选择阶段：X509CertificateCollection.AddRange时传入了null值，根源是单例ApiAdapter中的HttpClientHandler.ClientCertificates集合不是线程安全的。在并行请求的多线程环境下，并发修改或访问该集合会导致集合状态损坏，出现null证书元素，最终引发SSL连接失败。

解决办法 #

1. 初始化时一次性添加证书 #

将证书添加操作移到ApiAdapter构造函数中，确保证书只被添加一次，避免运行时并发修改ClientCertificates集合：

// 修改ApiAdapter构造函数
public ApiAdapter()
{
    _cookieContainer = new CookieContainer();
    _httpClientHandler = new HttpClientHandler
    {
        CookieContainer = _cookieContainer
    };
    // 初始化时添加证书，后续不再修改
    AddCertificates();
    _httpClient = new HttpClient(_httpClientHandler);
}

// 移除PostRequestAsync中操作handler证书的逻辑
public async Task<TResult> PostRequestAsync<TResult, TBodyRequest>(string testApiRelativeUrl, TBodyRequest bodyRequest,
            CancellationToken cancellationToken = default) where TResult : class
{
    var uriBuilder = BuildRequestUri(testApiRelativeUrl);
    var request = new HttpRequestMessage(HttpMethod.Post, uriBuilder.Uri);
    request.Content = new StringContent(_jsonSerializer.SerializeObject(bodyRequest), Encoding.UTF8, "application/json");
    var response = await _httpClient.SendAsync(request, cancellationToken);
    return await ExtractBodyFromResponseAsync<TResult>(response);
}

2. 控制并行请求的并发度 #

过高的并发会加剧线程安全问题，同时耗尽HTTP连接池。给Parallel.ForEachAsync设置合理的并发上限：

await Parallel.ForEachAsync(datasToProcess, new ParallelOptions { MaxDegreeOfParallelism = 10 }, async (_, token) =>
{
    var fakfurTestApiService = LoadTestKernel.Kernel.Get<IFakfurApiService>();
    var personCreated = await fakfurTestApiService.CallPostRequestAsync("<Url path>", <dataBody>);
});

3. 使用SocketsHttpHandler替代HttpClientHandler #

.NET Core及以后版本推荐使用SocketsHttpHandler，它的线程安全性和性能更优，能避免很多传统HttpClientHandler的线程安全问题：

// 修改ApiAdapter构造函数
public ApiAdapter()
{
    _cookieContainer = new CookieContainer();
    var httpHandler = new SocketsHttpHandler
    {
        CookieContainer = _cookieContainer,
        PooledConnectionLifetime = TimeSpan.FromMinutes(5)
    };
    // 初始化时添加证书
    if (_certificate2Collection != null)
    {
        httpHandler.SslOptions.ClientCertificates.AddRange(_certificate2Collection);
    }
    _httpClient = new HttpClient(httpHandler);
}

4. 为每个请求单独配置证书（动态场景） #

如果必须动态使用不同证书，不要修改全局HttpClientHandler的证书集合，而是为每个HttpRequestMessage单独配置：

public async Task<TResult> PostRequestAsync<TResult, TBodyRequest>(string testApiRelativeUrl, TBodyRequest bodyRequest,
            CancellationToken cancellationToken = default) where TResult : class
{
    var uriBuilder = BuildRequestUri(testApiRelativeUrl);
    var request = new HttpRequestMessage(HttpMethod.Post, uriBuilder.Uri);
    
    // 为当前请求单独添加证书
    if (_certificate2Collection != null)
    {
        foreach (var cert in _certificate2Collection)
        {
            request.ClientCertificates.Add(cert);
        }
    }
    
    request.Content = new StringContent(_jsonSerializer.SerializeObject(bodyRequest), Encoding.UTF8, "application/json");
    var response = await _httpClient.SendAsync(request, cancellationToken);
    return await ExtractBodyFromResponseAsync<TResult>(response);
}

内容的提问来源于stack exchange，提问作者huan le