Pandas如何去除DataFrame两列中反向重复的配对行

核心逻辑是为每一组关系生成不区分方向的唯一标识，再基于标识去重即可，不需要复杂的循环匹配。

具体步骤 #

• 对每一行的From和To字段值，按固定规则（比如字符串字典序）排序，组成不可变的元组作为无向边的唯一键：不管是(A,B)还是(B,A)，排序后生成的键完全一致
• 基于生成的唯一键做去重，保留每个键第一次出现的行
• 删除辅助用的键列，重置索引即可得到最终结果

实现代码 #

import pandas as pd

# 加载原始数据集
from_to_weight = {'From':['Dimitrios Rafailidis','Dimitrios Rafailidis','Dimitrios Rafailidis','Dimitrios Rafailidis','Dimitrios Rafailidis','Fabio Crestani','Fabio Crestani','Fabio Crestani','Tomasz Kusmierczyk','Tomasz Kusmierczyk','Tomasz Kusmierczyk','Kjetil Nørvåg','Kjetil Nørvåg','Simranjit Singh Sachar','Simranjit Singh Sachar','Simranjit Singh Sachar','Nicholas Diakopoulos','Nicholas Diakopoulos','Nicholas Diakopoulos','Nicholas Diakopoulos','Nasser Alsaedi', 'Nasser Alsaedi','Pete Burnap', 'Pete Burnap','Omer F. Rana','Omer F. Rana','Omer F. Rana','Omer F. Rana','Omer F. Rana'],
'To':['Fabio Crestani', 'Tomasz Kusmierczyk', 'Simranjit Singh Sachar', 'Nicholas Diakopoulos', 'Omer F. Rana', 'Dimitrios Rafailidis', 'Nicholas Diakopoulos', 'Omer F. Rana','Dimitrios Rafailidis','Kjetil Nørvåg', 'Simranjit Singh Sachar','Tomasz Kusmierczyk', 'Kjetil Nørvåg','Dimitrios Rafailidis','Tomasz Kusmierczyk','Nicholas Diakopoulos','Dimitrios Rafailidis','Fabio Crestani', 'Simranjit Singh Sachar', 'Omer F. Rana','Pete Burnap', 'Omer F. Rana','Nasser Alsaedi', 'Omer F. Rana','Dimitrios Rafailidis', 'Fabio Crestani', 'Nicholas Diakopoulos', 'Nasser Alsaedi', 'Pete Burnap'],
'weight':[2,1,1,1,1,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]}
from_to_weight_df = pd.DataFrame(data=from_to_weight)

# 生成无向边的唯一标识
from_to_weight_df['edge_key'] = from_to_weight_df.apply(
    lambda row: tuple(sorted([row['From'], row['To']])), axis=1
)

# 去重得到结果
result_df = from_to_weight_df.drop_duplicates(
    subset='edge_key', keep='first'
).drop(columns='edge_key').reset_index(drop=True)

运行代码后得到的result_df和给出的期望输出完全匹配，共15条记录，所有反向重复的配对都被剔除，且保留了首次出现的关系方向和对应权重。

注：如果业务场景需要把双向边的权重累加后保留，只需要在去重前按edge_key分组对weight求和即可，不需要改动核心的标识生成逻辑。

内容的提问来源于stack exchange，提问作者ArgyGr