ABP v4.4.3分层MVC项目调用_tenantStore.FindAsync触发TaskCanceledException异常

运行环境 #

• ABP Framework版本：v4.4.3
• UI类型：MVC
• DB提供器：EF Core
• 架构：分层MVC（Identity Server独立部署）

异常信息 #

日志显示遍历Volo.Saas.EntityFrameworkCore.SaasDbContext上下文的查询结果时触发异常，抛出System.Threading.Tasks.TaskCanceledException: 任务已取消，堆栈跟踪如下：

2021-11-03 21:21:43.774 -05:00 [ERR] An exception occurred while iterating over the results of a query for context type 'Volo.Saas.EntityFrameworkCore.SaasDbContext'.
System.Threading.Tasks.TaskCanceledException: A task was canceled.
at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.InitializeAsync(DbDataReader reader, IReadOnlyList`1 columns, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList`1 columns, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList`1 columns, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync() System.Threading.Tasks.TaskCanceledException: A task was canceled. at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.InitializeAsync(DbDataReader reader, IReadOnlyList`1 columns, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList`1 columns, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList`1 columns, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
2021-11-03 21:21:43.838 -05:00 [ERR] A task was canceled.
System.Threading.Tasks.TaskCanceledException: A task was canceled.
at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.InitializeAsync(DbDataReader reader, IReadOnlyList`1 columns, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList`1 columns, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList`1 columns, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync() at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken) at Volo.Abp.Domain.Repositories.EntityFrameworkCore.EfCoreRepository`3.FindAsync(TKey id, Boolean includeDetails, CancellationToken cancellationToken)
at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo)
at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue`1.ProceedAsync() at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation) at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter`1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func`3 proceed) at Volo.Saas.Tenants.TenantStore.GetCacheItemAsync(Nullable`1 id, String name)
at Volo.Saas.Tenants.TenantStore.FindAsync(Guid id)
at ClientsLink.Data.ClientsLinkTenantDatabaseMigrationHandler.MigrateAndSeedForTenantAsync(Guid tenantId, String adminEmail, String adminPassword) in D:\Century\Internal\Clients.Link\Clients.Link\src\ClientsLink.Domain\Data\ClientsLinkTenantDatabaseMigrationHandler.cs:line 98

异常触发代码位置如下：

private async Task MigrateAndSeedForTenantAsync(
    Guid tenantId,
    string adminEmail,
    string adminPassword)
{
  try
  {
    using (_currentTenant.Change(tenantId))
    {
      // Create database tables if needed
      using (var uow = _unitOfWorkManager.Begin(requiresNew: true, isTransactional: false))
      {
        var tenantConfiguration = await _tenantStore.FindAsync(tenantId); // 此处抛出异常
        if (tenantConfiguration?.ConnectionStrings != null &&
            !tenantConfiguration.ConnectionStrings.Default.IsNullOrWhiteSpace())
        {
          foreach (var migrator in _dbSchemaMigrators)
          {
            _logger.LogDebug($"Running migration {migrator.ToString()}");
            await migrator.MigrateAsync();
          }
        }

        await uow.CompleteAsync();
      }

复现步骤 #

1. 使用ABP Commercial Suite v4.4.3生成分层MVC示例应用
2. 新增两个实体：1个全局宿主实体，1个多租户实体，且配置多租户实体指向全局实体的导航属性
3. 创建示例数据库
4. 运行DbMigrator，宿主数据库迁移完成
5. 启动Web应用，用admin账号登录后新建测试租户，约1分钟后UI返回内部错误，日志上报上述异常

问题描述 #

经两周排查确认异常由_tenantStore.FindAsync(tenantId)调用被阻塞导致，已尝试以下操作均未解决问题：

• 更换所有依赖组件版本
• 两台不同设备上重新构建项目
• 重写数据种子贡献器
• 将System.Data.SqlClient替换为Microsoft.Data.SqlClient，同时将连接字符串的Command Timeout参数设置为3600
  需要明确该调用超时或被取消的可能原因。

可能原因 #

1. 租户上下文切换错误：你在调用_tenantStore.FindAsync之前已经通过_currentTenant.Change(tenantId)切换到了目标租户上下文，而TenantStore默认是从宿主数据库读取租户配置信息的，切换租户上下文后会自动路由到租户数据库执行查询，此时租户数据库尚未完成迁移创建，查询无法建立连接最终超时被取消。这是该场景下最高概率的触发原因，调整逻辑为先查询租户配置，再切换租户上下文执行迁移即可验证。
2. 工作单元取消令牌绑定上层请求：你开启的新工作单元默认会继承上层请求的取消令牌，前端新建租户的请求如果触发了服务端默认的请求超时（通常是60秒），或者用户主动断开页面，上层取消令牌会直接终止当前工作单元内的所有异步操作，和数据库命令超时配置无关。可以在开启工作单元时传入CancellationToken.None验证是否属于该情况。
3. 导航属性配置导致无效查询：多租户实体关联全局实体的导航属性配置有误，EF Core生成查询语句时生成了跨库Join的无效SQL，或者触发了全表扫描，导致查询长时间执行被取消。可以开启EF Core的SQL日志，查看_tenantStore.FindAsync实际执行的SQL语句是否符合预期。
4. 查询过滤器冲突：自定义的全局查询过滤器和ABP自带的多租户、软删除等过滤器冲突，导致查询租户表时添加了错误的过滤条件，出现锁等待或全表扫描，长时间执行后触发取消。
5. 租户缓存与锁冲突：新建租户操作会对SaasTenant表加行锁，TenantStore如果缓存配置失效，查询直接穿透到数据库，会出现锁等待，超过超时时间后被取消。

内容的提问来源于stack exchange，提问作者jlavallet