解决Sklearn中ValueError：无法将字符串转换为float的问题

CSV文件结构 #

我的CSV文件共1054行，结构如下：

lemma,trained
iran seizes bitcoin mining machines power spike,-1
...

我的Python代码 #

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
from sklearn.metrics import accuracy_score
from sklearn.naive_bayes import GaussianNB
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

df = pd.read_csv('lemma copy.csv')
X = df.iloc[:, 0].values
y = df.iloc[:, 1].values
print(y)

X_train, X_test, y_train, y_test =train_test_split(X,y,test_size= 0.25, random_state=0)

sc_X = StandardScaler() 

X_train = sc_X.fit_transform(X_train)

运行错误信息 #

运行代码时触发以下错误：

Traceback (most recent call last):
  File "/home/arctesian/Scripts/School/EE/Algos/Qual/bayes/sklean.py", line 20, in <module>
    X_train = sc_X.fit_transform(X_train)
  File "/home/arctesian/.local/lib/python3.10/site-packages/sklearn/base.py", line 867, in fit_transform
    return self.fit(X, **fit_params).transform(X)
  File "/home/arctesian/.local/lib/python3.10/site-packages/sklearn/preprocessing/_data.py", line 809, in fit
    return self.partial_fit(X, y, sample_weight)
  File "/home/arctesian/.local/lib/python3.10/site-packages/sklearn/preprocessing/_data.py", line 844, in partial_fit
    X = self._validate_data(
  File "/home/arctesian/.local/lib/python3.10/site-packages/sklearn/base.py", line 577, in _validate_data
    X = check_array(X, input_name="X", **check_params)
  File "/home/arctesian/.local/lib/python3.10/site-packages/sklearn/utils/validation.py", line 856, in check_array
    array = np.asarray(array, order=order, dtype=dtype)
ValueError: could not convert string to float: 'twitter ios beta lays groundwork bitcoin tips'

排查后确认是随机拆分后文本无法转换为数值导致的问题，请问该如何解决？

你这问题本质是**StandardScaler只处理数值型数据，而你的X是纯文本，根本没法直接转成浮点数**，必须先把文本转换成机器学习能识别的数值特征，再做后续处理。另外，针对文本分类任务，GaussianNB不是最佳选择，换成MultinomialNB（多项式朴素贝叶斯）适配性更好，下面是修正后的完整代码：

具体步骤 #

1. 文本转数值特征：用TfidfVectorizer做TF-IDF提取，这是文本分类里常用的方法，能把每段文本转换成衡量词语重要性的数值矩阵。
2. 替换适配的分类器：MultinomialNB专门针对离散的文本特征设计，比高斯贝叶斯更适合这类任务。

修正后的完整代码 #

import numpy as np
import pandas as pd
from sklearn.metrics import accuracy_score
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split

# 读取数据
df = pd.read_csv('lemma copy.csv')
X = df['lemma'].values
y = df['trained'].values

# 拆分数据集
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)

# 将文本转换为TF-IDF数值特征
tfidf = TfidfVectorizer()
X_train = tfidf.fit_transform(X_train)
X_test = tfidf.transform(X_test)

# 使用多项式朴素贝叶斯训练模型
model = MultinomialNB()
model.fit(X_train, y_train)

# 预测并输出准确率
y_pred = model.predict(X_test)
print(f"模型准确率: {accuracy_score(y_test, y_pred):.2f}")

额外说明 #

• TF-IDF本身已经是标准化后的特征（取值在0-1之间），完全不需要再用StandardScaler做缩放，反而会打乱特征的原有分布。
• 如果非要坚持使用GaussianNB，需要把TF-IDF生成的稀疏矩阵转成稠密矩阵（效率会有所降低），代码调整如下：

# 将稀疏矩阵转换为稠密矩阵（仅适配GaussianNB）
X_train_dense = X_train.toarray()
X_test_dense = X_test.toarray()

model = GaussianNB()
model.fit(X_train_dense, y_train)

内容的提问来源于stack exchange，提问作者Daniel Okita