调用APICallAsync时频繁触发TaskCanceledException的原因排查

问题现象 #

项目中多处调用APICallAsync方法时，每1000次调用约有100次会抛出System.Threading.Tasks.TaskCanceledException异常。

异常堆栈信息 #

System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<<SendCoreAsync>g__Core|5_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<<SendCoreAsync>g__Core|5_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at AirlineConnector.OmanAirways.Helpers.Helper.<ProviderCallAsync>d__12.MoveNext()A task was canceled.-   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<<SendCoreAsync>g__Core|5_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<<SendCoreAsync>g__Core|5_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at AirlineConnector.OmanAirways.Helpers.Helper.<ProviderCallAsync>d__12.MoveNext()

相关代码实现 #

using System;
using System.Linq;
using System.Net;
using System.Net.Http;
using Microsoft.Extensions.DependencyInjection;
using System.Text;

namespace Helpers
{
    public class Helper 
    {
        private static readonly Lazy<IHttpClientFactory> httpClientFactoryLazy = new Lazy<IHttpClientFactory>(() =>
        {
            var services = new ServiceCollection();
            services.AddHttpClient("HttpClientFactory", client =>
            {
                client.Timeout = TimeSpan.FromSeconds(15);
            })
              .ConfigurePrimaryHttpMessageHandler(() =>
              {
                  var handler = new HttpClientHandler();
                  handler.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
                  return handler;
              });
            var serviceProvider = services.BuildServiceProvider();
            return serviceProvider.GetRequiredService<IHttpClientFactory>();
        });

        private static IHttpClientFactory httpClientFactory
        {
            get
            {
                return httpClientFactoryLazy.Value;
            }
        }
        public async Task<string> APICallAsync(string remoteRequest, string serviceURL, string ActionURL)
        {
            var rawResponse = "";
            try
            {
                HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, serviceURL);
                request.Content = new StringContent(remoteRequest, Encoding.UTF8, "text/xml");
                request.Headers.Add("Accept-Encoding", "gzip, deflate");
                request.Headers.Add("SOAPAction", ActionURL);

                ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12 | System.Net.SecurityProtocolType.Tls11 | System.Net.SecurityProtocolType.Tls;
              
                var _httpClient = httpClientFactory.CreateClient("HttpClientFactory");

                HttpResponseMessage response = await _httpClient.SendAsync(request);
                rawResponse = await response?.Content?.ReadAsStringAsync();
            }
            catch (Exception ex)
            {
            }
            return rawResponse;
        }
    }
}

可能的原因分析 #

• 超时配置不匹配服务实际响应能力：当前HttpClient设置的15秒超时阈值，若目标服务在高峰时段或部分请求场景下处理耗时超过该值，就会触发任务取消。10%的失败率说明服务端响应稳定性不足，需要结合实际请求耗时统计调整超时时间。
• HttpClientFactory的隔离性导致连接池失效：代码中手动创建独立的ServiceCollection和ServiceProvider，而非复用应用全局的DI容器。这会使得每个Helper类使用的HttpClientFactory是独立实例，无法共享连接池资源，高并发场景下容易出现连接耗尽，进而引发请求超时或被取消。
• HttpRequestMessage未正确释放资源：每次调用都新建HttpRequestMessage，但未使用using语句包裹以自动释放资源，长期高调用量下可能导致内存或句柄泄漏，间接引发请求异常。
• 全局安全协议配置时机错误：在每次请求中重复设置ServicePointManager.SecurityProtocol，该配置是全局生效的，重复设置不仅无意义，还可能在并发场景下引发线程安全问题，影响TLS连接的正常建立。
• 异常吞杀导致问题定位困难：catch块直接捕获所有异常但不做任何日志记录或处理，无法区分是超时、网络错误还是其他类型的异常，也无法为后续排查提供有效信息。

内容的提问来源于stack exchange，提问作者Nishad