使用EFCore.BulkExtensions批量Upsert超大数据时遇tempdb空间不足及性能问题的求助
我现在用EFCore.BulkExtensions写了一个批量Upsert的扩展方法,但是处理200万条数据的时候,执行时间居然要17分钟,到后面还直接抛出了异常:
Could not allocate space for object 'dbo.SORT temporary run storage: 140737501921280' in database 'tempdb' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.
The transaction log for database 'tempdb' is full due to 'ACTIVE_TRANSACTION' and the holdup lsn is (41:136:347)
执行过程中我能看到C分区的可用空间一直在减少(截图显示C盘可用空间随操作进行持续下降)。重启SQL Server之后,C盘的可用空间又回到了30GB左右。我试过用多线程并行插入,但执行时间没什么明显变化。想请教下大家有没有什么优化建议,或者我写的代码是不是有什么问题?
注:循环处理实体设置CreatedDate和UpdatedDate的部分哪怕是200万条数据也没花多少时间,问题应该不在这。
public static async Task<OperationResultDto> AddOrUpdateBulkByTransactionAsync<TEntity>(this DbContext _myDatabaseContext, List<TEntity> data) where TEntity : class { using (var transaction = await _myDatabaseContext.Database.BeginTransactionAsync()) { try { _myDatabaseContext.Database.SetCommandTimeout(0); var currentTime = DateTime.Now; // Disable change tracking _myDatabaseContext.ChangeTracker.AutoDetectChangesEnabled = false; // Set CreatedDate and UpdatedDate for each entity foreach (var entity in data) { var createdDateProperty = entity.GetType().GetProperty("CreatedDate"); if (createdDateProperty != null && (createdDateProperty.GetValue(entity) == null || createdDateProperty.GetValue(entity).Equals(DateTime.MinValue))) { // Set CreatedDate only if it's not already set createdDateProperty.SetValue(entity, currentTime); } var updatedDateProperty = entity.GetType().GetProperty("UpdatedDate"); if (updatedDateProperty != null) { updatedDateProperty.SetValue(entity, currentTime); } } // Bulk insert or update var updateByProperties = GetUpdateByProperties<TEntity>(); var bulkConfig = new BulkConfig() { UpdateByProperties = updateByProperties, CalculateStats = true, SetOutputIdentity = false }; // Batch size for processing int batchSize = 50000; for (int i = 0; i < data.Count; i += batchSize) { var batch = data.Skip(i).Take(batchSize).ToList(); await _myDatabaseContext.BulkInsertOrUpdateAsync(batch, bulkConfig); } // Commit the transaction if everything succeeds await transaction.CommitAsync(); return new OperationResultDto { OperationResult = bulkConfig.StatsInfo }; } catch (Exception ex) { // Handle exceptions and roll back the transaction if something goes wrong transaction.Rollback(); return new OperationResultDto { Error = new ErrorDto { Details = ex.Message + ex.InnerException?.Message } }; } finally { // Re-enable change tracking _myDatabaseContext.ChangeTracker.AutoDetectChangesEnabled = true; } } }
备注:内容来源于stack exchange,提问作者Abdulaziz Burghal

