Python中过滤停用词统计词频的嵌套for循环效率优化问题

原代码核心问题 #

• 逻辑错误：嵌套循环的写法会导致每个非停用词被重复计数，最终得到的词频结果完全不符合实际
• 时间复杂度过高：嵌套循环带来*O(n²)*的时间复杂度，单词量越大耗时呈指数级上涨
• 停用词查询效率低：用列表存储停用词，in查询需要遍历整个列表，单次查询耗时为O(m)（m为停用词数量）

最优实现方案 #

我们可以通过3个改动把耗时降到毫秒级：

1. 把停用词转成集合，in查询耗时降到O(1)
2. 去掉嵌套循环，单次遍历即可完成过滤+计数
3. 直接用Python标准库collections.Counter完成词频统计，底层为C实现效率远高于手写循环

优化后代码如下：

from collections import Counter

words_cleaned = [...] # 莎士比亚戏剧的单词列表
# 停用词转集合提升查询效率
stop_words = {"i", "me", "my", "myself", "we", "our", "ours", "ourselves", "you", "your", "yours", "yourself", "yourselves", "he", "him", "his", "himself", "she", "her", "hers", "herself", "it", "its", "itself", "they", "them", "their", "theirs", "themselves", "what", "which", "who", "whom", "this", "that", "these", "those", "am", "is", "are", "was", "were", "be", "been", "being", "have", "has", "had", "having", "do", "does", "did", "doing", "a", "an", "the", "and", "but", "if", "or", "because", "as", "until", "while", "of", "at", "by", "for", "with", "about", "against", "between", "into", "through", "during", "before", "after", "above", "below", "to", "from", "up", "down", "in", "out", "on", "off", "over", "under", "again", "further", "then", "once", "here", "there", "when", "where", "why", "how", "all", "any", "both", "each", "few", "more", "most", "other", "some", "such", "no", "nor", "not", "only", "own", "same", "so", "than", "too", "very", "s", "t", "can", "will", "just", "don", "should", "now"}

# 一行完成过滤+统计
go_word_counts = Counter(word for word in words_cleaned if word not in stop_words)

无额外依赖的原生实现 #

如果不想引入collections库，也可以用原生字典实现，性能同样远高于原代码：

stop_words = {"i", "me", "my", "myself", "we", "our", "ours", "ourselves", "you", "your", "yours", "yourself", "yourselves", "he", "him", "his", "himself", "she", "her", "hers", "herself", "it", "its", "itself", "they", "them", "their", "theirs", "themselves", "what", "which", "who", "whom", "this", "that", "these", "those", "am", "is", "are", "was", "were", "be", "been", "being", "have", "has", "had", "having", "do", "does", "did", "doing", "a", "an", "the", "and", "but", "if", "or", "because", "as", "until", "while", "of", "at", "by", "for", "with", "about", "against", "between", "into", "through", "during", "before", "after", "above", "below", "to", "from", "up", "down", "in", "out", "on", "off", "over", "under", "again", "further", "then", "once", "here", "there", "when", "where", "why", "how", "all", "any", "both", "each", "few", "more", "most", "other", "some", "such", "no", "nor", "not", "only", "own", "same", "so", "than", "too", "very", "s", "t", "can", "will", "just", "don", "should", "now"}
go_word_counts = {}

for word in words_cleaned:
    if word not in stop_words:
        go_word_counts[word] = go_word_counts.get(word, 0) + 1

两种优化方案的时间复杂度都是O(n)，你原场景16秒的耗时优化后基本可以降到0.1秒以内，且得到的词频结果是正确的。

内容的提问来源于stack exchange，提问作者zelfde