Python词频统计代码用in判断无法完全移除无意义停用词问题求助

问题原因 #

• 遍历动态修改的列表导致漏检：你在直接遍历file_contents单词列表的同时调用remove()方法删除元素，列表删除元素后后续所有元素的索引会自动前移，for循环的遍历指针还是按原有索引顺序后移，就会跳过被删元素后面紧邻的那个元素。如果出现连续的停用词，第二个停用词根本不会被检查到，自然就会大量残留。
• （可选优化点）停用词用列表做in判断时间复杂度是O(n)，数据量大的时候效率较低，建议改成集合类型。

修复后的代码 #

def calculate_frequencies(file_contents):
    punctuations = '''!()-[]{};:'"\,<>./?@#$%^&*_~'''
    # 停用词转集合，查找效率提升到O(1)
    uninteresting_words = {"the", "a", "to", "if", "is", "it", "of", "and", "or", "an", "as", "i", "me", "my", 
    "we", "our", "ours", "you", "your", "yours", "he", "she", "him", "his", "her", "hers", "its", "they", "them", 
    "their", "what", "which", "who", "whom", "this", "that", "am", "are", "was", "were", "be", "been", "being", 
    "have", "has", "had", "do", "does", "did", "but", "at", "by", "with", "from", "here", "when", "where", "how", 
    "all", "any", "both", "each", "few", "more", "some", "such", "no", "nor", "too", "very", "can", "will", "just"}
    
    # 过滤标点
    for punctuation in punctuations:
         file_contents = file_contents.replace(punctuation,"")
    
    file_contents = file_contents.lower()
    word_list = file_contents.split()
    
    # 用列表推导式生成过滤后的新列表，不修改原遍历对象，避免漏检
    filtered_words = [word for word in word_list if word not in uninteresting_words]
    
    # 统计词频
    word_count = {}
    for word in filtered_words:
        word_count[word] = word_count.get(word, 0) + 1
    return word_count

# 调用示例，替换为你实际读取的文件内容即可
# with open('你的文本文件.txt', 'r', encoding='utf-8') as f:
#     content = f.read()
# print(calculate_frequencies(content))

额外说明 #

原代码最后返回的是print(file_contents)的执行结果，实际值为None，如果你需要拿到词频统计结果，直接返回统计好的word_count字典即可。

内容的提问来源于stack exchange，提问作者freekp