Tf-IDF向量化适配MultinomialNB遇NaN错误，阿拉伯文本被误判为空

问题背景 #

预处理含两列的数据集（正负标签列+阿拉伯语推文列）后，运行训练代码时出现「输入包含NaN」的错误，已尝试data.dropna(inplace=True)但阿拉伯语字符串被误判为null值，更换代码逻辑后问题仍存在。

初始训练代码 #

X_train, X_test, y_train, y_test = train_test_split(data['Tweet'], data['Label'], test_size=0.20, random_state=0)
tf_idf = TfidfVectorizer()

x_traintf = tf_idf.fit_transform(X_train)
x_traintf = tf_idf.transform(X_train)
x_testtf = tf_idf.fit_transform(X_test)
x_testtf = tf_idf.transform(X_test)

naive_bayes_classifier = MultinomialNB()
naive_bayes_classifier.fit(x_traintf, y_train)
y_pred = naive_bayes_classifier.predict(x_testtf)
print(metrics.classification_report(y_test, y_pred, target_names=['pos', 'neg']))

错误信息 #

Traceback (most recent call last):
  File "C:\Users\Lenovo\PycharmProjects\pythonProject1\main.py", line 72, in <module>
    naive_bayes_classifier.fit(x_traintf, y_train)
  File "C:\Users\Lenovo\PycharmProjects\pythonProject1\venv\Lib\site-packages\sklearn\naive_bayes.py", line 749, in fit
    X, y = self._check_X_y(X, y)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lenovo\PycharmProjects\pythonProject1\venv\Lib\site-packages\sklearn\naive_bayes.py", line 583, in _check_X_y
    return self._validate_data(X, y, accept_sparse="csr", reset=reset)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lenovo\PycharmProjects\pythonProject1\venv\Lib\site-packages\sklearn\base.py", line 565, in _validate_data
    X, y = check_X_y(X, y, **check_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lenovo\PycharmProjects\pythonProject1\venv\Lib\site-packages\sklearn\utils\validation.py", line 1122, in check_X_y
    y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Lenovo\PycharmProjects\pythonProject1\venv\Lib\site-packages\sklearn\utils\validation.py", line 1144, in _check_y
    _assert_all_finite(y, input_name="y", estimator_name=estimator_name)
  File "C:\Users\Lenovo\PycharmProjects\pythonProject1\venv\Lib\site-packages\sklearn\utils\validation.py", line 111, in _assert_all_finite
    raise ValueError("Input contains NaN")
ValueError: Input contains NaN

尝试的其他代码（仍报错） #

X_train, X_test, y_train, y_test = train_test_split(all_X, all_y, test_size=0.2, random_state=42) 
vectorizer = TfidfVectorizer()
X_train_tfidf = vectorizer.fit_transform(X_train)
clf = MultinomialNB() 
clf.fit(X_train_tfidf, y_train)
X_test_tfidf = vectorizer.transform(X_test)
y_pred = clf.predict(X_test_tfidf)
print(metrics.accuracy_score(y_test, y_pred))

解决方案 #

1. 修复阿拉伯语文本的NaN误判问题 #

Pandas默认的dropna()仅识别标准NaN/None值，阿拉伯语文本中的空白字符串、全角空格等会被误判为空。需先统一清理无效值：

import numpy as np

# 将空白/全角空格等替换为标准NaN
data['Tweet'] = data['Tweet'].replace(r'^\s*$', np.nan, regex=True)
data['Label'] = data['Label'].replace(r'^\s*$', np.nan, regex=True)

# 指定删除Tweet和Label列的NaN行
data = data.dropna(subset=['Tweet', 'Label'])

# 验证清理结果
print(data.isnull().sum())

# 额外过滤标签列，确保仅保留有效标签值
data = data[data['Label'].isin(['pos', 'neg'])]

2. 修正TF-IDF的错误使用 #

初始代码中对测试集调用fit_transform()是错误的——测试集必须使用训练集拟合的向量器转换，否则会导致特征空间不一致：

X_train, X_test, y_train, y_test = train_test_split(data['Tweet'], data['Label'], test_size=0.20, random_state=0)
tf_idf = TfidfVectorizer()

# 仅对训练集执行fit_transform，测试集只用transform
x_traintf = tf_idf.fit_transform(X_train)
x_testtf = tf_idf.transform(X_test)

naive_bayes_classifier = MultinomialNB()
naive_bayes_classifier.fit(x_traintf, y_train)
y_pred = naive_bayes_classifier.predict(x_testtf)
print(metrics.classification_report(y_test, y_pred, target_names=['pos', 'neg']))

3. 阿拉伯语适配优化 #

针对阿拉伯语的词法特征，调整TF-IDF参数提升效果：

tf_idf = TfidfVectorizer(
    token_pattern=r'\b\w+\b',  # 适配阿拉伯语单词分词
    encoding='utf-8',  # 确保编码正确识别阿拉伯语
    # 若有阿拉伯语停用词库，可导入后设置stop_words参数，如nltk的arabic_stopwords
)

内容的提问来源于stack exchange，提问作者Tareq Ewaida