C#如何提升30万+数据量大型for循环的执行速度？

核心问题定位 #

你的代码性能瓶颈不在于循环本身的并行度，而是循环内重复遍历全量业务列表的时间复杂度过高：30万条结果，每条都要遍历订单、分享、招聘等多个列表做筛选计算，整体时间复杂度达到O(N*M)级别，并行只能利用多核，解决不了算法层面的冗余计算问题。

优化步骤 #

1. 预构建索引字典消除循环内遍历 #

把从数据库拉取的所有业务数据，提前按查询条件分组聚合，存为O(1)查询的字典结构，避免循环内重复筛选。

示例代码：

// 1. 先预构建storeUsers的索引字典，优化初始list生成
var storeUserDict = storeUsers
    .GroupBy(x => x.StoreUser.StoreUserRadixCode)
    .ToDictionary(g => g.Key, g => g.First().StoreUser);

// 2. 预聚合订单数据
var orderDict = order_list
    .GroupBy(x => new { x.StoreUserRadixCode, x.CreateTime.Date })
    .ToDictionary(g => g.Key, g => new 
    {
        RealSale = g.Sum(x => x.RealPrice),
        Num = g.Sum(x => x.OrderNumber),
        OrderNum = g.Count(),
        Sale = g.Sum(x => x.OrderAmount),
        ReturnOrderNum = g.Sum(x => x.ReturnOrderNumber),
        Refund = g.Sum(x => x.ReturnOrderAmount)
    });

// 3. 预聚合分享访问数据
var shareVisitDict = share_visit_list
    .GroupBy(x => new { x.StoreUserRadixCode, x.CreateTime.Date })
    .ToDictionary(g => g.Key, g => new 
    {
        PV = g.Count(),
        UV = g.Select(x => x.OpenId).Distinct().Count()
    });

// 4. 预聚合招聘数据：外层key是用户编码，内层key是日期，同时预存总数
var recruitDict = recruit_list
    .GroupBy(x => x.StoreUserRadixCode)
    .ToDictionary(g => g.Key, g => new 
    {
        Total = g.Count(),
        DateCount = g.GroupBy(x => x.CreateTime.Date).ToDictionary(dg => dg.Key, dg => dg.Count())
    });

// 5. 预聚合外部联系人数据
var externalDict = external_list
    .GroupBy(x => x.StoreUserHisId)
    .ToDictionary(g => g.Key, g => new 
    {
        Total = g.Count(),
        DateCount = g.GroupBy(x => x.CreateTime.Date).ToDictionary(dg => dg.Key, dg => dg.Count())
    });

// 6. 预聚合公众号拉新数据
var recruitMpDict = recruit_mp_list
    .GroupBy(x => x.StoreUserHisId)
    .ToDictionary(g => g.Key, g => new 
    {
        Total = g.Count(),
        DateCount = g.GroupBy(x => x.CreateTime.Date).ToDictionary(dg => dg.Key, dg => dg.Count())
    });

// 7. 预聚合关系数据
var relationDict = relation_his_list
    .Where(x => x.ShareBindTime.HasValue)
    .GroupBy(x => new { x.ShareBy, x.ShareBindTime.Value.Date })
    .ToDictionary(g => g.Key, g => g.Count());

2. 优化初始列表生成逻辑 #

原来的生成逻辑每次循环都要遍历storeUsers查找用户，换成预构建的字典直接取值，时间复杂度从O(N*M)降到O(N)：

var date_diff = endTime.Value.Subtract(startTime.Value).Days;
// 提前指定容量减少扩容损耗
var list = new List<DailyUserStaticticeModel>(date_diff * storeUserRadixCodes.Count); 
for (var i = 0; i < date_diff; i++)
{
    var currentDate = startTime.Value.AddDays(i).Date;
    foreach (var radixCode in storeUserRadixCodes)
    {
        list.Add(new DailyUserStaticticeModel
        {
            CurrentDate = currentDate,
            StoreUser = storeUserDict[radixCode]
        });
    }
}

3. 重写结果赋值逻辑，全部使用字典查询 #

优化后循环内所有取值都是O(1)操作，30万条数据遍历耗时可以降到毫秒级：

foreach (var item in list)
{
    var currentDate = item.CurrentDate.Date;
    var radixCode = item.StoreUser.StoreUserRadixCode;
    var userId = item.StoreUser.Id;

    // 订单数据取值
    var orderKey = new { StoreUserRadixCode = radixCode, Date = currentDate };
    if (orderDict.TryGetValue(orderKey, out var orderStat))
    {
        item.RealSale = orderStat.RealSale;
        item.Num = orderStat.Num;
        item.OrderNum = orderStat.OrderNum;
        item.Sale = orderStat.Sale;
        item.ReturnOrderNum = orderStat.ReturnOrderNum;
        item.Refund = orderStat.Refund;
    }

    // 分享访问数据取值
    var shareKey = new { StoreUserRadixCode = radixCode, Date = currentDate };
    if (shareVisitDict.TryGetValue(shareKey, out var shareStat))
    {
        item.PagePV = shareStat.PV;
        item.PageUV = shareStat.UV;
    }

    // 招聘数据取值
    if (recruitDict.TryGetValue(radixCode, out var recruitStat))
    {
        item.RecruitmentTotal = recruitStat.Total;
        recruitStat.DateCount.TryGetValue(currentDate, out var recruitDayCount);
        item.RecruitmentNum = recruitDayCount;
    }

    // 外部联系人数据取值
    if (externalDict.TryGetValue(userId, out var externalStat))
    {
        item.QyExternalTotal = externalStat.Total;
        externalStat.DateCount.TryGetValue(currentDate, out var externalDayCount);
        item.QyExternalNum = externalDayCount;
    }

    // 公众号拉新数据取值
    if (recruitMpDict.TryGetValue(userId, out var mpStat))
    {
        item.RecruitMPTotal = mpStat.Total;
        mpStat.DateCount.TryGetValue(currentDate, out var mpDayCount);
        item.RecruitMPNum = mpDayCount;
    }

    // 关系数据取值
    var relationKey = new { ShareBy = radixCode, Date = currentDate };
    relationDict.TryGetValue(relationKey, out var relationCount);
    item.RelationHisNum = relationCount;
}

4. 可选优化点 #

如果优化后性能还有提升需求，可以再做以下调整：

• 数据库查询层面优化：添加ShopGuideBrandId、BrandID、CreateTime等字段的联合索引，减少数据拉取耗时
• 所有列表预初始化容量，避免GC扩容损耗
• 若数据量极大，可以把上述循环替换为Parallel.ForEach，此时所有读操作都是线程安全的，不会有并发问题
• 日期相关的计算统一提前处理成Date类型，避免循环内重复调用.Date属性

内容的提问来源于stack exchange，提问作者oppoovo