自定义KMeans与CountVectorizer构建BERTopic时fit_transform报NaN错误

现象 #

调用fit_transform(final_df.body)时先出现除零警告：

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/bertopic/vectorizers/_ctfidf.py:69: RuntimeWarning:
divide by zero encountered in divide

随后抛出ValueError：

ValueError                                Traceback (most recent call last)
<ipython-input-104-1f024d22018f> in <module>
----> 1 topics, probs = bert_topic_model.fit_transform(final_df.body)

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/bertopic/_bertopic.py in fit_transform(self, documents, embeddings, y)
    368             self._map_representative_docs(original_topics=True)
    369         else:
--> 370             self._save_representative_docs(documents)
    371 
    372         self.probabilities_ = self._map_probabilities(probabilities, original_topics=True)

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/bertopic/_bertopic.py in _save_representative_docs(self, documents)
   3000                 bow = self.vectorizer_model.transform(selected_docs)
   3001                 ctfidf = self.ctfidf_model.transform(bow)
--> 3002                 sim_matrix = cosine_similarity(ctfidf, self.c_tf_idf_[topic + self._outliers])
   3003 
   3004                 # Extract top 3 most representative documents

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sklearn/metrics/pairwise.py in cosine_similarity(X, Y, dense_output)
   1178     # to avoid recursive import
   1179 
--> 1180     X, Y = check_pairwise_arrays(X, Y)
   1181 
   1182     X_normalized = normalize(X, copy=True)

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sklearn/utils/validation.py in inner_f(*args, **kwargs)
     61             extra_args = len(args) - len(all_args)
     62             if extra_args <= 0:
--> 63                 return f(*args, **kwargs)
     64 
     65             # extra_args > 0

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sklearn/metrics/pairwise.py in check_pairwise_arrays(X, Y, precomputed, dtype, accept_sparse, force_all_finite, copy)
    144                             estimator=estimator)
    145         else:
--> 146         X = check_array(X, accept_sparse=accept_sparse, dtype=dtype,
    147                         copy=copy, force_all_finite=force_all_finite,
    148                         estimator=estimator)

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sklearn/utils/validation.py in inner_f(*args, **kwargs)
     61             extra_args = len(args) - len(all_args)
     62             if extra_args <= 0:
--> 63                 return f(*args, **kwargs)
     64 
     65             # extra_args > 0

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sklearn/utils/validation.py in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator)
    648     if sp.issparse(array):
    649         _ensure_no_complex_data(array)
--> 650         array = _ensure_sparse_format(array, accept_sparse=accept_sparse,
    651                                       dtype=dtype, copy=copy,
    652                                       force_all_finite=force_all_finite,

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sklearn/utils/validation.py in _ensure_sparse_format(spmatrix, accept_sparse, dtype, copy, force_all_finite, accept_large_sparse)
    446                           % spmatrix.format, stacklevel=2)
    447         else:
--> 448             _assert_all_finite(spmatrix.data,
    449                                allow_nan=force_all_finite == 'allow-nan')
    450 
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sklearn/utils/validation.py in _assert_all_finite(X, allow_nan, msg_dtype)
    101                 not allow_nan and not np.isfinite(X).all()):
    102             type_err = 'infinity' if allow_nan else 'NaN, infinity'
--> 103             raise ValueError(
    104                     msg_err.format
    105                     (type_err,

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

复现代码 #

features = final_df["body"] # 已确认无NaN或无穷大值
transformerVectoriser = CountVectorizer(analyzer = 'word', ngram_range = (1, 4), vocabulary = vocab_list)
# 已确认vocab_list无NaN或无穷大值，均为字符串

cluster_model = KMeans(n_clusters = 50, init='k-means++', max_iter = 1500, random_state=None)

bert_topic_model = BERTopic(hdbscan_model = cluster_model, 
                            vectorizer_model = transformerVectoriser, 
                            verbose = True, 
                            top_n_words = 15)

# 已确认final_df.body无NaN或无穷大值，均为字符串
topics, probs = bert_topic_model.fit_transform(final_df.body) # 报错位置

排查与解决方案 #

1. 处理零向量文档与词汇表匹配问题 #

• 核心原因：自定义vocab_list中的词汇在部分文档（或聚类后的某类文档）中完全未出现，导致CountVectorizer生成全零词袋，后续CTF-IDF计算触发除零，产生NaN/inf。
• 解决步骤：
  • 过滤空文档：确保所有文档非空

    final_df = final_df[final_df.body.str.strip() != ""]
    

  • 验证词汇表覆盖率：移除从未在文本中出现的词汇

    from collections import Counter
    all_words = " ".join(final_df.body).split()
    word_counts = Counter(all_words)
    valid_vocab = [word for word in vocab_list if word_counts.get(word, 0) > 0]
    transformerVectoriser = CountVectorizer(analyzer='word', ngram_range=(1,4), vocabulary=valid_vocab)
    

  • 添加min_df参数：过滤仅在极少数文档中出现的词汇，减少零向量概率

    transformerVectoriser = CountVectorizer(analyzer='word', ngram_range=(1,4), vocabulary=valid_vocab, min_df=2)
    

2. 修复CTF-IDF计算的除零问题 #

• 核心原因：BERTopic默认CTF-IDF未开启平滑处理，当某类文档词频总和为0时触发除零。
• 解决方法：自定义CTF-IDF模型并开启平滑

  from bertopic.vectorizers import ClassTfidfTransformer
  ctfidf_model = ClassTfidfTransformer(smooth_idf=True, normalize=True)
  bert_topic_model = BERTopic(hdbscan_model=cluster_model,
                              vectorizer_model=transformerVectoriser,
                              ctfidf_model=ctfidf_model,
                              verbose=True,
                              top_n_words=15)
  

  smooth_idf会在计算时添加平滑项，避免除零操作。

3. 适配聚类模型参数 #

• 核心原因：KMeans聚类数设置不合理，可能导致某聚类中全是零向量文档。
• 解决步骤：
  • 提前检查词袋矩阵的稀疏情况：

    bow = transformerVectoriser.transform(final_df.body)
    # 查看全零向量文档数量
    zero_docs = (bow.sum(axis=1) == 0).sum()
    print(f"全零向量文档数量：{zero_docs}")
    

  • 若存在零向量文档，先过滤后再聚类；若聚类数不合理，可通过肘部法则确定合适的n_clusters值。

内容的提问来源于stack exchange，提问作者taga