如何合并同结构DataFrame并基于多列去除重复数据

操作步骤 #

• 用pd.concat()合并两个列结构一致的DataFrame
• 调用drop_duplicates()方法，指定关键列完成去重

完整代码实现 #

import pandas as pd

data1 = {'first_column':  ['1', '2', '2', '2'],
        'second_column': ['1', '2', '2', '2'],
       'key_column1':['1', '2', '2', '6'],
      'key_column2':['1', '2', '2', '1'],
       'fourth_column':['1', '2', '2', '2'],
        }
df1 = pd.DataFrame(data1)

data2 = {'first_column':  ['1', '2', '2', '2'],
        'second_column': ['1', '2', '2', '2'],
       'key_column1':['1', '3', '2', '6'],
      'key_column2':['1', '5', '2', '2'],
       'fourth_column':['1', '2', '2', '2'],
        }
df2 = pd.DataFrame(data2)

# 合并两个DataFrame并重置索引
merged_df = pd.concat([df1, df2], ignore_index=True)

# 按指定关键列去重，保留首次出现的记录
deduped_df = merged_df.drop_duplicates(subset=['key_column1', 'key_column2'], keep='first')

print(deduped_df)

最终输出 #

内容的提问来源于stack exchange，提问作者yoopiyo