.NET 5 ASP.NET应用中System.InvalidCastException异常排查求助

问题描述 #

在.NET 5的ASP.NET Web应用中，每日会出现几次System.InvalidCastException异常，无法按需复现。异常发生时通常会连续出现多次，导致ASP.NET Core应用崩溃；IIS重启应用后可正常运行数小时或数天，直到再次触发。

该应用每日处理超15万次请求，此问题相对罕见，约每5-20万次请求出现一次，暂未发现引发异常的请求规律，仍在排查中。

异常堆栈跟踪 #

An unhandled exception has occurred while executing the request.
System.InvalidCastException: Unable to cast object of type 'System.Data.SqlClient.SqlTransaction' to type 'System.Transactions.BucketSet'.
   at System.Transactions.TransactionTable.AddIter(InternalTransaction txNew)
   at System.Transactions.TransactionTable.Add(InternalTransaction txNew)
   at System.Transactions.InternalTransaction..ctor(TimeSpan timeout, CommittableTransaction committableTransaction)
   at System.Transactions.CommittableTransaction..ctor(IsolationLevel isoLevel, TimeSpan timeout)
   at System.Transactions.TransactionScope..ctor(TransactionScopeOption scopeOption, TransactionOptions transactionOptions, TransactionScopeAsyncFlowOption asyncFlowOption)
   at MyApp.Shared.TransactionHandling.GetTransactionScope(Int32 timeOutMinutes) in C:\TeamCity\buildAgent\work\fd104265242f4376\server\MyApp.Shared\TransactionHandling.cs:line 34
   at MyApp.Shared.TransactionHandling.Run[T](Func`1 action) in C:\TeamCity\buildAgent\work\fd104265242f4376\server\MyApp.Shared\TransactionHandling.cs:line 85
   at MyApp.BL.Access.AccessGroupService.Create(Int32 planId, List`1 accessGroups) in C:\TeamCity\buildAgent\work\fd104265242f4376\server\MyApp.BL\Access\AccessGroupService.cs:line 241
   at MyApp.SL.Administration.Controllers.AccessGroupsController.PostAccessGroupsAsync(Int32 planId) in C:\TeamCity\buildAgent\work\fd104265242f4376\server\MyApp.SL.Api\Controllers\Administration\AccessGroupsController.cs:line 43
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at MyApp.SL.Api.StoreTenantForLoggingMiddleware.InvokeAsync(HttpContext httpContext, TenantContext tenantContext) in C:\TeamCity\buildAgent\work\fd104265242f4376\server\MyApp.SL.Api\StoreTenantForLoggingMiddleware.cs:line 43
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

相关代码实现 #

AccessGroupService类 #

public class MyApp.BL.Access.AccessGroupService
{ 
    // ... 其他代码省略 ... 

    public IEnumerable<ExpandoObject> Create(int planId, List<ExpandoObject> accessGroups)
        => TransactionHandling.Run(() =>
        {
              // ... 执行大量EF Core调用逻辑 ...
        }
}

TransactionHandling静态类 #

public static class TransactionHandling
{
    private static int _maxTimeoutMinutes = 10; // 系统默认值

    public static T Run<T>(Func<T> action)
    {
        using (var scope = GetTransactionScope())
        {
            var result = action();
            scope.Complete();
            return result;
        }
    }

    public static TransactionScope GetTransactionScope(int timeOutMinutes = 0)
    {
        var transactionOptions = new TransactionOptions
        {
            IsolationLevel = IsolationLevel.ReadCommitted
        };

        // 特殊超时处理，但异常发生时timeOutMinutes大多为0，暂不认为与此有关
        if (timeOutMinutes > 0)
        {
            // 参考相关问题解决方案，仅当超时值大于当前最大值时更新全局设置
            transactionOptions.Timeout = TimeSpan.FromMinutes(timeOutMinutes);

            if (timeOutMinutes > _maxTimeoutMinutes)
            {
                _maxTimeoutMinutes = timeOutMinutes;
                SetTransactionManagerField("s_cachedMaxTimeout", true);
                SetTransactionManagerField("s_maximumTimeout", transactionOptions.Timeout);
            }
        }

        return new TransactionScope(TransactionScopeOption.Required, transactionOptions, TransactionScopeAsyncFlowOption.Enabled);
    }

    private static void SetTransactionManagerField(string fieldName, object value)
    {
        typeof(TransactionManager).GetField(fieldName, BindingFlags.NonPublic | BindingFlags.Static).SetValue(null, value);
    }

    // ... 其他代码省略 ... 

}

排查进展与推测 #

• 查看TransactionTable的源码实现，未发现直接导致异常的原因；初步猜测是SqlClient对象跨线程使用不当，代码库同时使用EF Core和ADO.NET，但尚未定位到具体问题点。
• 应用还偶发System.InvalidOperationException: Internal .NET Framework Data Provider error 3异常，堆栈信息关联Hangfire，怀疑是ADO对象多线程访问导致，计划将Hangfire改用Microsoft.Data.SqlClient进行测试。
• 更新1：发现ExpandoObject的深克隆操作可能引发WeakReference相关问题，异常触发的方法中确实使用了基于Stack Overflow某实现的深克隆逻辑，怀疑这是问题成因，后续会反馈排查进展。

内容的提问来源于stack exchange，提问作者Rory