使用Sklearn CountVectorizer过滤低频特征报lower not found错误求助

原有代码如下：

K = 10
XYtr['description'] = XYtr['description'].fillna("nan")
Xte['description'] = Xte['description'].fillna("nan")
corpus = list(XYtr['description'])+list(Xte['description'])
vectorizer = CountVectorizer()
corpus = vectorizer.fit_transform(corpus)
lda = LatentDirichletAllocation(n_components = K)
lda.fit(corpus)
# 到此处为止代码运行无问题

# 构建按词频排序的（词项，词频）元组列表
sum_words = corpus.sum(axis=0) 
words_freq = [(word, sum_words[0, idx]) for word, idx in vectorizer.vocabulary_.items()]
words_freq = sorted(words_freq, key = lambda x: x[1])

# 仅保留词项存入列表
vocabulary, _ = zip(*words_freq[:int(total_features * 0.2)])
vocabulary = list(vocabulary)

# 最终使用该词表将模型限制为仅处理低频词
bottom_vect = CountVectorizer(vocabulary=vocabulary)
topics = bottom_vect.fit_transform(corpus)

上述代码最后一行运行时报错AttributeError: lower not found，无法获取topics变量结果。

报错原因 #

第一次调用CountVectorizer.fit_transform()时，原本存储原始文本的corpus变量被覆盖为了稀疏矩阵类型。后续新建的bottom_vect需要接收原始文本作为输入，它默认会对输入内容执行小写转换操作，数值类型的稀疏矩阵没有lower方法，因此触发报错。同时原有代码未定义total_features变量，也会触发后续报错。

修复后代码 #

K = 10
XYtr['description'] = XYtr['description'].fillna("nan")
Xte['description'] = Xte['description'].fillna("nan")
# 保留原始文本语料
corpus = list(XYtr['description'])+list(Xte['description'])
vectorizer = CountVectorizer()
# 向量化结果赋值给新变量
corpus_vec = vectorizer.fit_transform(corpus)
lda = LatentDirichletAllocation(n_components = K)
lda.fit(corpus_vec)

# 构建按词频排序的（词项，词频）元组列表
sum_words = corpus_vec.sum(axis=0) 
words_freq = [(word, sum_words[0, idx]) for word, idx in vectorizer.vocabulary_.items()]
words_freq = sorted(words_freq, key = lambda x: x[1])
# 补充定义总特征数
total_features = len(vectorizer.vocabulary_)

# 仅保留词项存入列表
vocabulary, _ = zip(*words_freq[:int(total_features * 0.2)])
vocabulary = list(vocabulary)

# 最终使用该词表将模型限制为仅处理低频词
bottom_vect = CountVectorizer(vocabulary=vocabulary)
# 传入原始文本语料完成向量化
topics = bottom_vect.fit_transform(corpus)

数据集示例 #

• XYtr数据集：
  
• Xte数据集：
  

内容的提问来源于stack exchange，提问作者stanley cho