使用KMeans聚类推特文本时遇ValueError等异常求助

问题 #

尝试对包含已清洗推特（存储于text_clean列）的数据集应用无监督学习，执行KMeans的fit(X)操作时，出现AttributeError: 'NoneType' object has no attribute 'split'以及ValueError: setting an array element with a sequence错误。已对数据集做了空值处理，相关代码及报错栈如下：

完整代码 #

import pandas as pd
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt

# 加载CSV数据
df = pd.read_csv("nombre_del_archivo.csv")

# 删除含空值的行
df.dropna(inplace=True)

# 填充text_clean列剩余空值为空字符串
df['text_clean'].fillna('', inplace=True)

# 文本向量化
vectorizer = TfidfVectorizer(stop_words='english')
X = vectorizer.fit_transform(df['text_clean'])

# 肘部法寻找最优聚类数
wcss = []
for i in range(2, 11):
    kmeans = KMeans(n_clusters=i, init='k-means++', max_iter=300, n_init=10, random_state=0)
    kmeans.fit(X)
    wcss.append(kmeans.inertia_)

报错代码段 #

wcss = []
for i in range(2, 11):
    kmeans = KMeans(n_clusters=i, init='k-means++', max_iter=300, n_init=10, random_state=0)
    kmeans.fit(X)
    wcss.append(kmeans.inertia_)
    
plt.plot(range(2, 11), wcss)
plt.title('肘部法')
plt.xlabel('聚类数')
plt.ylabel('WCSS')
plt.show()

完整报错栈 #

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
TypeError: float() argument must be a string or a number, not 'csr_matrix'

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
<ipython-input-46-f1e874523304> in <module>
      3 for i in range(2, 11):
      4     kmeans = KMeans(n_clusters=i, init='k-means++', max_iter=300, n_init=10, random_state=0)
----> 5     kmeans.fit(X)
      6     wcss.append(kmeans.inertia_)
      7 

~\anaconda3\lib\site-packages\sklearn\cluster\_kmeans.py in fit(self, X, y, sample_weight)
   1415         self._validate_params()
   1416 
-> 1417         X = self._validate_data(
   1418             X,
   1419             accept_sparse="csr",

~\anaconda3\lib\site-packages\sklearn\base.py in _validate_data(self, X, y, reset, validate_separately, **check_params)
    544             raise ValueError("Validation should be done on X, y or both.")
    545         elif not no_val_X and no_val_y:
--> 546             X = check_array(X, input_name="X", **check_params)
    547             out = X
    548         elif no_val_X and not no_val_y:

~\anaconda3\lib\site-packages\sklearn\utils\validation.py in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
    877                     array = xp.astype(array, dtype, copy=False)
    878                 else:
--> 879                     array = _asarray_with_order(array, order=order, dtype=dtype, xp=xp)
    880             except ComplexWarning as complex_warning:
    881                 raise ValueError(

~\anaconda3\lib\site-packages\sklearn\utils\_array_api.py in _asarray_with_order(array, dtype, order, copy, xp)
    183     if xp.__name__ in {"numpy", "numpy.array_api"}:
    184         # Use NumPy API to support order
--> 185         array = numpy.asarray(array, order=order, dtype=dtype)
    186         return xp.asarray(array, copy=copy)
    187     else:

ValueError: setting an array element with a sequence.

已确认数据中无NoneType或空值，求解决方法。

解决方法 #

1. 统一text_clean列的数据类型 #

报错核心是TfidfVectorizer处理时遇到非字符串类型数据，即便填充了空值，列中可能存在数值、列表等非字符串对象。先验证数据类型：

print(df['text_clean'].apply(type).unique())

若输出包含非str类型，统一转为字符串并清理异常值：

df['text_clean'] = df['text_clean'].astype(str)
# 替换识别为字符串的空值标记
df['text_clean'] = df['text_clean'].replace(['nan', 'None'], '')

2. 处理稀疏矩阵兼容性问题 #

TfidfVectorizer.fit_transform()返回CSR稀疏矩阵，部分旧版本Scikit-learn对其支持有bug，可转为密集矩阵（注意：大数据量会占用大量内存）：

X = vectorizer.fit_transform(df['text_clean']).toarray()

或升级依赖库到最新稳定版：

pip install --upgrade scikit-learn numpy pandas

3. 过滤无效文本 #

添加步骤过滤空字符串或仅含空白字符的行：

# 过滤空文本
df = df[df['text_clean'].str.strip() != '']
# 重置索引避免后续索引问题
df.reset_index(drop=True, inplace=True)

4. 适配KMeans参数版本 #

若使用旧版Scikit-learn，n_init参数在新版本中默认值已调整，确保参数符合当前版本要求，升级库可避免此类参数冲突。

内容的提问来源于stack exchange，提问作者Alba