Pandas DataFrame逐行分组列表时如何优化慢循环代码

现有存储列表对的Pandas DataFrame：包含Type、Name两列，每个单元格存储长度相等的列表，同一行内两个列表的元素一一对应，形成(Type, Name)值对。需要实现两个目标：

• 按每行内Type的取值，对对应的Name元素分组
• 提取所有Type的唯一值作为新列名，每个新列存储对应行中、匹配该Type值的Name元素组成的列表

原有实现采用嵌套for循环配合iterrows()逐行遍历赋值，运行效率极低，以下是可复现的低效示例代码：

import pandas as pd
df = pd.DataFrame({"Type": [["1", "1", "2", "3"], ["2","3"]], "Name": [["A", "B", "C", "D"], ["E", "F"]]})

unique = list(set(row["Type"]))
for t in unique:
    df[t] = None
    df[t] = df[t].astype('object')

for idx, row in df.iterrows():
    for t in unique:
        df.at[idx, t] = [row["Name"][i] for i in range(len(row["Name"])) if row["Type"][i] == t]

核心优化逻辑：彻底规避iterrows逐行转Series、循环内反复触发pandas索引/类型检查的开销，用原生Python逻辑完成行内分组，最后批量构造新列拼接，数据量越大速度优势越明显，比原有实现快10~100倍。

通用版本（兼顾可读性与速度） #

import pandas as pd
from collections import defaultdict

df = pd.DataFrame({
    "Type": [["1", "1", "2", "3"], ["2","3"]],
    "Name": [["A", "B", "C", "D"], ["E", "F"]]
})

# 提取全量唯一Type值，可按需排序
all_types = sorted({t for type_list in df["Type"] for t in type_list})

def group_row(type_list, name_list):
    group_map = defaultdict(list)
    for t, n in zip(type_list, name_list):
        group_map[t].append(n)
    return [group_map.get(t, []) for t in all_types]

# 批量生成所有新列数据
new_cols = list(zip(*[group_row(tl, nl) for tl, nl in zip(df["Type"], df["Name"])]))
# 一次性赋值所有新列
for t, col_values in zip(all_types, new_cols):
    df[t] = col_values

极致性能版本（适配百万行以上大数据量） #

完全跳过pandas的apply、逐单元格赋值逻辑，直接用原生Python遍历生成新列数据后整体拼接，性能达到最优：

import pandas as pd
from collections import defaultdict

df = pd.DataFrame({
    "Type": [["1", "1", "2", "3"], ["2","3"]],
    "Name": [["A", "B", "C", "D"], ["E", "F"]]
})

all_types = sorted({t for type_list in df["Type"] for t in type_list})
new_col_data = []

for type_list, name_list in zip(df["Type"], df["Name"]):
    group_map = defaultdict(list)
    for t, n in zip(type_list, name_list):
        group_map[t].append(n)
    new_col_data.append([group_map.get(t, []) for t in all_types])

# 直接构造新列DataFrame后拼接，无逐行逐单元格赋值开销
df = pd.concat([
    df,
    pd.DataFrame(new_col_data, columns=all_types)
], axis=1)

运行后得到的结果如下：

Type                  Name        1    2    3
0  [1, 1, 2, 3]  [A, B, C, D]  [A, B]  [C]  [D]
1        [2, 3]        [E, F]       []  [E]  [F]

• iterrows()遍历过程中会将每一行转换为Series对象，本身遍历开销是原生Python列表遍历的几十倍
• 循环内反复调用df.at[idx, t]赋值，每次操作都会触发pandas的索引校验、类型校验，重复开销极高
• 每行对每个Type值都单独做一次列表遍历，重复遍历次数多，时间复杂度冗余

内容的提问来源于stack exchange，提问作者Jason