按行迭代优化：改用列表/字典能否提升大数据处理速度？

我现在需要按行迭代表格计算数值并生成结果DataFrame，但大数据量下这个过程慢得离谱。想请教：如果把循环里的资源选取、工时更新操作的数据源从DataFrame改成列表的列表或者字典的列表，能不能提升处理速度？

简化示例代码如下：

import pandas as pd
import heapq
case_list=[]
Operations = pd.DataFrame({'operation_id': [1, 2, 3, 4], 'Work': [4, 5, 6, 4]})
Machines = pd.DataFrame({'operation_id': [2, 2, 1, 1, 3, 4], 'MachineID': [4, 5, 6, 4, 5, 6], 'MinusTimeFundMachine': [-4, -5, -6, 0, -5,-6]})
Workers = pd.DataFrame({'operation_id': [2, 2, 1, 1, 3, 4], 'MachineID': [4, 5, 6, 4, 5, 6], 'worker': [4, 5, 6, 4, 5, 6], 'MinusTimeFundSotr': [-4, -5, -6, -4, -5,-6]})
print(Operations)
print(Machines)
print(Workers)
for ix in Operations.index:

    #Choosing resources
    dct = Machines[Machines['operation_id'] == Operations.loc[ix]['operation_id']][
        ["MinusTimeFundMachine", "MachineID"]].values.tolist()
    heapq.heapify(dct)
    MinusTimeFundMachine, MachineID = heapq.heappop(dct)

    dct2 = Workers[(Workers['operation_id'] == Operations.loc[ix]['operation_id']) & (Workers.MachineID == MachineID)][
        ["MinusTimeFundSotr", "worker"]].values.tolist()
    heapq.heapify(dct2)
    MinusTimeFundSotr2, worker = heapq.heappop(dct2)


    #Changing working time funds
    Workers.loc[Workers.worker == worker, 'MinusTimeFundSotr'] = Workers.loc[
                                                                    Workers.worker == worker, 'MinusTimeFundSotr'] + Operations.loc[ix]['Work']
    Machines.loc[Machines.MachineID == MachineID, 'MinusTimeFundMachine'] = Machines.loc[
                                                                        Machines.MachineID == MachineID, 'MinusTimeFundMachine'] + Operations.loc[ix]['Work']



    case_list.append({'Operation': Operations.loc[ix]['operation_id'], 'MachineID': MachineID, 'worker': worker})

print()
print(pd.DataFrame.from_dict(case_list))

结论：改用列表/字典列表确实能显著提速 #

DataFrame的行级索引、切片操作本身存在固定开销，在循环中反复执行时，这些开销会被持续放大。换成原生Python数据结构（列表的列表、字典的列表）后，能彻底避开DataFrame的内部机制损耗，大幅提升循环内操作的效率。

具体优化思路 #

1. 预转换数据结构：把Machines和Workers提前转换成索引化的字典结构，避免每次循环都做DataFrame切片：
   • 将Machines按operation_id分组，存储为{op_id: [(MinusTimeFundMachine, MachineID), ...]}格式
   • 将Workers按(operation_id, MachineID)组合键分组，存储为{(op_id, machine_id): [(MinusTimeFundSotr, worker), ...]}格式
2. 用字典跟踪实时状态：用字典记录机器和工人的当前工时，替代循环中修改DataFrame的操作——字典的查找和更新都是O(1)级别的高效操作，远快于DataFrame的loc方法。

优化后的代码示例 #

import pandas as pd
import heapq

# 原始数据
Operations = pd.DataFrame({'operation_id': [1, 2, 3, 4], 'Work': [4, 5, 6, 4]})
Machines = pd.DataFrame({'operation_id': [2, 2, 1, 1, 3, 4], 'MachineID': [4, 5, 6, 4, 5, 6], 'MinusTimeFundMachine': [-4, -5, -6, 0, -5,-6]})
Workers = pd.DataFrame({'operation_id': [2, 2, 1, 1, 3, 4], 'MachineID': [4, 5, 6, 4, 5, 6], 'worker': [4, 5, 6, 4, 5, 6], 'MinusTimeFundSotr': [-4, -5, -6, -4, -5,-6]})

# 预转换Machines为字典：key=operation_id，value=[(MinusTimeFundMachine, MachineID), ...]
machine_map = {}
for _, row in Machines.iterrows():
    op_id = row['operation_id']
    item = (row['MinusTimeFundMachine'], row['MachineID'])
    machine_map.setdefault(op_id, []).append(item)

# 预转换Workers为字典：key=(operation_id, MachineID)，value=[(MinusTimeFundSotr, worker), ...]
worker_map = {}
for _, row in Workers.iterrows():
    key = (row['operation_id'], row['MachineID'])
    item = (row['MinusTimeFundSotr'], row['worker'])
    worker_map.setdefault(key, []).append(item)

# 用字典跟踪实时工时，替代修改DataFrame
machine_time = {row['MachineID']: row['MinusTimeFundMachine'] for _, row in Machines.iterrows()}
worker_time = {row['worker']: row['MinusTimeFundSotr'] for _, row in Workers.iterrows()}

case_list = []

# 循环处理每个操作
for _, op_row in Operations.iterrows():
    op_id = op_row['operation_id']
    work_hours = op_row['Work']
    
    # 选择机器：从预存列表取数据堆化
    machine_items = machine_map[op_id].copy()
    heapq.heapify(machine_items)
    MinusTimeFundMachine, MachineID = heapq.heappop(machine_items)
    
    # 选择工人：根据op_id和机器ID取对应数据堆化
    worker_key = (op_id, MachineID)
    worker_items = worker_map[worker_key].copy()
    heapq.heapify(worker_items)
    MinusTimeFundSotr2, worker = heapq.heappop(worker_items)
    
    # 更新工时：直接操作字典
    worker_time[worker] += work_hours
    machine_time[MachineID] += work_hours
    
    case_list.append({'Operation': op_id, 'MachineID': MachineID, 'worker': worker})

# 输出结果
print(pd.DataFrame.from_dict(case_list))

额外优化建议 #

• 若数据量极大，可尝试用numpy数组替代列表，进一步提升堆操作和数据访问的效率
• 所有预处理逻辑都放在循环外完成，彻底避免循环内的DataFrame操作
• 若业务逻辑允许，优先尝试将循环逻辑向量化（用Pandas内置函数替代循环），这是Pandas场景下的最优提速方案

内容的提问来源于stack exchange，提问作者Daneel Ank