如何去除pandas DataFrame Keywords列跨行列的不区分大小写重复关键词

实现思路 #

• 先对所有关键词做标准化处理：按逗号分割后去除前后空白、转小写，统计每个标准化关键词对应的出现行数
• 逐行过滤关键词，仅保留只在1行中出现过的关键词，保留原始关键词的大小写和格式后拼接返回

完整代码 #

import pandas as pd
from collections import defaultdict

# 构造测试数据，你可以替换成自己的DataFrame读取逻辑
data = {
    "Code": ["A", "B", "C", "D"],
    "Keywords": [
        "Real estate, loan, building, office, land, warehouse",
        "Real Estate Lease , Real Estate, building, Office, Warehouse, rental, Tenant, broker advisor, Real Estate Lease , Lease and rent",
        "Transport Air freight, shift, cargo, truck, insurance, Transport Insurance, Transport",
        "Transport, shift, cargo, truck, insurance, Transport Insurance"
    ]
}
df = pd.DataFrame(data)

# 统计每个标准化关键词的出现行数
keyword_row_cnt = defaultdict(int)
for kw_content in df["Keywords"]:
    # 同一行重复出现的关键词只统计1次行数
    current_row_kws = set(kw.strip().lower() for kw in kw_content.split(","))
    for std_kw in current_row_kws:
        keyword_row_cnt[std_kw] += 1

# 过滤每行的关键词
def filter_keywords(kw_content):
    original_kws = kw_content.split(",")
    filtered = [kw for kw in original_kws if keyword_row_cnt[kw.strip().lower()] == 1]
    # 整理格式后返回
    return ", ".join(kw.strip() for kw in filtered) if filtered else ""

df["Keywords"] = df["Keywords"].apply(filter_keywords)

# 输出结果
print(df)

逻辑调整说明 #

如果你的需求是只要关键词全局出现次数≥2就删除，不管重复出现在同一行还是不同行，只需要把统计逻辑替换为下方代码即可：

# 统计每个标准化关键词的全局出现次数
keyword_total_cnt = defaultdict(int)
for kw_content in df["Keywords"]:
    for kw in kw_content.split(","):
        std_kw = kw.strip().lower()
        keyword_total_cnt[std_kw] += 1

后续过滤时判断keyword_total_cnt[kw.strip().lower()] == 1即可。 #

内容的提问来源于stack exchange，提问作者petre_isp