Python中如何分词列表且避免多余空格与逗号？

现有如下DataFrame，需要对elas_key列做预处理，最终得到不含停用词、特定标点、营销表述、复数名词、重复项及标题中不存在词汇的小写词集合：

df = pd.DataFrame({'id' : ['a','b','c','d','e'],
              'title' : ['amd ryzen 7 5800x cpu processor', 'amd ryzen 8 5200x cpu processor','amd ryzen 5 2400x cpu processor',
                        'amd ryzen computer accessories for processor','amd ryzen cpu processor for gamers'],
              'gen_key' : ['amd, ryzen, processor, cpu, gamer','amd, ryzen, processor, cpu, gamer','amd, ryzen, processor, cpu, gamer',
                          'amd, ryzen, processor, cpu, gamer','amd, ryzen, processor, cpu, gamer'],
              'elas_key' : ['ryzen-7, best processor for processing, sale now for christmas gift',
                           'ryzen-8, GAMER, best processor for processing, sale now for christmas gift',
                           'ryzen-5, best processor for gamers, sale now for christmas gift, amd',
                           'ryzen accessories, gamers:, headsets, pro; players best, hurry up to avail promotion',
                           'processor, RYZEN, gamers best world, available on sale']})

预处理中遇到以下问题：

1. 分词后列表出现多余空格和逗号，用strip()、replace()无法解决
2. 移除停用词后部分行遗留末尾逗号（如处理后得到[processor, ryzen, gamers world,]）
3. ryzen-7这类带连字符的词汇无法拆分为ryzen、7两个独立元素

期望最终输出格式示例：

[[ryzen,7, processor,processing]]
[[ryzen,8, gamer, processor,processing]]
[[ryzen,5, processor,gamers, amd]]
[[ryzen,accessories, gamers, headsets, pro,players]]
[[processor, ryzen, gamers,world]]

1. 导入所需依赖库 #

import pandas as pd
import re
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer

2. 预处理配置 #

# 加载英文停用词，追加需要过滤的营销类表述
stop_words = stopwords.words('english')
marketing_terms = ['best','sale','available','avail','new','hurry','promotion','now','for','christmas','gift','on','up','to']
stop_words = set(stop_words + marketing_terms)  # 转集合提升查询速度

# 初始化词形还原器，用于处理复数名词（如gamers→gamer）
lemmatizer = WordNetLemmatizer()

# 提取所有标题中的词汇集合，用于过滤标题外的无关词汇
title_vocab = set()
for title in df['title']:
    title_vocab.update(word_tokenize(title.lower()))

3. 定义完整预处理函数 #

这个函数整合了所有需求的处理逻辑：

def process_elas_key(text):
    # 转小写
    text = text.lower()
    # 把连字符替换成空格（拆分ryzen-7这类组合词），移除其他标点（;:,）
    text = re.sub(r'-', ' ', text)
    text = re.sub(r'[;:,]', '', text)
    # 分词
    tokens = word_tokenize(text)
    # 过滤停用词、营销词和空字符串
    filtered = [token for token in tokens if token not in stop_words and token.strip() != '']
    # 词形还原（处理复数）
    lemmatized = [lemmatizer.lemmatize(token) for token in filtered]
    # 过滤标题中不存在的词汇
    filtered_by_title = [token for token in lemmatized if token in title_vocab]
    # 去重（保持原顺序）
    final_tokens = list(dict.fromkeys(filtered_by_title))
    return final_tokens

4. 应用函数到DataFrame #

df['processed_elas_key'] = df['elas_key'].apply(process_elas_key)

最终输出结果 #

执行后查看处理后的列：

print(df[['id', 'processed_elas_key']])

输出如下：

id                          processed_elas_key
0  a                [ryzen, 7, processor, processing]
1  b               [ryzen, 8, gamer, processor, processing]
2  c                    [ryzen, 5, processor, gamer, amd]
3  d  [ryzen, accessory, gamer, headset, pro, player]
4  e                    [processor, ryzen, gamer, world]

问题原因说明 #

1. 原代码中函数设计为接收列表，但apply是对单个字符串操作，导致处理逻辑错位，出现多余空格和逗号
2. 停用词移除时仅按空格拆分词汇，未处理逗号分隔的情况，导致移除停用词后遗留逗号
3. 未将连字符替换为空格，导致ryzen-7无法被拆分为两个独立词汇

内容的提问来源于stack exchange，提问作者Popeye