Pandas分组后标签列重复删除问题：NLP作业中pandas列重复字符串去重求助

问题根因 #

你现有代码去重失效的核心原因是：聚合标签时使用, （逗号+空格）作为拼接分隔符，但后续拆分标签时使用了默认的空格作为拆分符，导致拆分得到的标签末尾自带逗号，和实际标签值不匹配，无法识别重复项。

最优解决方案：分组聚合阶段直接去重 #

无需聚合后再二次处理，分组时直接完成去重，效率更高：

import pandas as pd

# 加载示例数据
data = {'ID2': {0: '440', 1: '440', 2: '440', 3: '440', 4: '422', 5: '2422', 6: '422', 
7: '422', 8: '422', 9: '422', 10: '422', 11: '422', 12: '422', 13: '422', 14: '422', 
15: '422', 16: '422', 17: '422', 18: '422', 19: '422', 20: '422', 21: '422', 22: '422'}, 
'comment': {0: 'prompt', 1: 'prompt', 2: 'prompt', 3: 'prompt', 4: 'prompt', 
5: 'prompt', 6: 'prompt', 7: 'great service', 8: 'great service', 9: 'great service', 
10: 'friendly', 11: 'friendly', 12: 'friendly', 13: 'friendly', 14: 'fairly organized', 
15: 'fairly organized', 16: 'fairly organized', 17: 'fairly organized',
18: 'fairly organized', 19: 'fairly organized', 20: 'fairly organized',
21: 'fairly organized', 22: 'fairly organized'}, 
'tags': {0: 'sp', 1: 'sp', 2: 'in', 3: 'ps', 4: 'wr', 5: 'sa', 6: 'sa', 7: 'sp', 
8: 'gs', 9: 'po', 10: 'av', 11: 'hf', 12: 'cs', 13: 'fr', 14: 'gs', 15: 'ly', 
16: 'drt', 17: 'co', 18: 'sp', 19: 'na', 20: 'ps', 21: 'ti', 22: 'ti'}}
docs = pd.DataFrame(data)

# 分组去重聚合：需要保留标签第一次出现的顺序用dict.fromkeys，不需要顺序可直接替换为set(x)
docs = docs.groupby(['ID2', 'comment'], as_index=False).agg(
    tags=('tags', lambda x: ', '.join(dict.fromkeys(x).keys()))
)

聚合后二次处理方案 #

如果已经完成了聚合操作，只需修改拆分逻辑的分隔符即可：

docs['new_tags'] = docs['tags'].str.split(', ').apply(lambda x: ', '.join(dict.fromkeys(x).keys()))

内容的提问来源于stack exchange，提问作者GSA