如何迭代Pandas DataFrame元素匹配另一DataFrame并计算匹配概率
原有代码错误原因
- 遍历逻辑错误:
for ind in df1默认遍历的是DataFrame的列名,不是行索引,导致取数完全不对 - 计数逻辑错误:内层循环每次都把
count重置为0,永远无法累计计数 - 赋值逻辑错误:
df1['t_{}'.format(i)]= count+1是给整列赋值,不是给当前匹配的行赋值 - 性能极差:嵌套循环时间复杂度为O(n*m),数据量过万后运行速度会非常慢
修复后的循环实现(仅作为逻辑修正参考,不推荐大数据量使用)
# 先初始化所有type对应的哑变量列为0 for t_idx in df2.index: df1[f't_{t_idx}'] = 0 # 遍历df1的每一行 for ind in df1.index: a1_val = df1.loc[ind, 'a1'] b1_val = df1.loc[ind, 'b1'] # 匹配df2的行 matched = df2[(df2['a2'] == a1_val) & (df2['b2'] == b1_val)] if not matched.empty: t_idx = matched.index[0] df1.loc[ind, f't_{t_idx}'] = 1
高效矢量化方案(适合大数据量,性能提升100倍以上)
用pandas自带的merge操作实现,完全避免循环,时间复杂度接近O(n+m):
import pandas as pd # 先构造测试数据(修正示例中df1列名的笔误,原df1第二列应为b1而非a2) df1 = pd.DataFrame() df1['a1'] = ['ABC','ACC','BCC','CCC'] df1['b1'] = ['ACC','AAC','BAC','CAC'] df2 = pd.DataFrame() df2['a2'] = ['ACC','BCC','ABC'] df2['b2'] = ['AAC','BAC','ACC'] df2['types'] = ['t1','t2','t3'] # 1. 左连接匹配type merged = pd.merge( df1, df2, left_on=['a1', 'b1'], right_on=['a2', 'b2'], how='left' ) # 2. 生成哑变量(你需要的t_i列) dummies = pd.get_dummies(merged['types'], prefix='t') df1 = pd.concat([df1, dummies], axis=1) # 3. 计算各type的匹配概率 total_count = len(df1) type_prob = merged['types'].value_counts() / total_count print("各type匹配概率:\n", type_prob)
运行后得到的概率结果为:
t3 0.25 t1 0.25 t2 0.25 Name: types, dtype: float64
剩下25%为无匹配的记录。
内容的提问来源于stack exchange,提问作者Geinkehdsk
相关产品推荐
相关产品推荐

