如何将SpaCy生成的文本向量输入Random Forest模型？

问题场景 #

将SpaCy生成的文本向量化数组与标签编码后的字段一起输入Random Forest分类器时，触发以下错误：

TypeError: only size-1 arrays can be converted to Python scalars
ValueError: setting an array element with a sequence.

用户原始代码如下：

数据构造代码 #

import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

d = {'le1': [0,1,2,1], 'le2': [3,0,2,1], 'spacy_output':[[0.12,0.14,3.5],[1.21,0.84,1.92],[0.34,0.85,2.43],[0.09,0.18,2.21]], 'response':[0,1,1,0]}

df = pd.DataFrame(d)

报错的模型训练代码 #

X = np.array(df.drop('response', axis=1))
y = df['response'].values.ravel()
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 23)

clf = RandomForestClassifier(min_samples_split=4, n_estimators=100, criterion='entropy')
clf.fit(X_train,y_train)

错误原因 #

spacy_output列存储的是嵌套列表，而sklearn模型要求特征矩阵必须是二维数值数组，每个元素必须是单一标量值，不能包含序列类型数据。

解决步骤 #

1. 拆分SpaCy向量化数组为独立特征列 #

把spacy_output中的每个向量元素拆成单独的特征列，合并到原DataFrame中：

# 展开spacy_output的向量为多列
spacy_features = pd.DataFrame(df['spacy_output'].tolist(), columns=[f'spacy_{i}' for i in range(len(df['spacy_output'][0]))])
# 合并到原数据并删除原始嵌套列
df_processed = pd.concat([df.drop('spacy_output', axis=1), spacy_features], axis=1)

2. 正常训练模型 #

处理后的DataFrame所有列都是标量数值，可直接输入模型：

X = df_processed.drop('response', axis=1).values
y = df_processed['response'].values.ravel()

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=23)

clf = RandomForestClassifier(min_samples_split=4, n_estimators=100, criterion='entropy')
clf.fit(X_train, y_train)

# 验证预测功能
y_pred = clf.predict(X_test)

完整修正代码 #

import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# 原始数据构造
d = {'le1': [0,1,2,1], 'le2': [3,0,2,1], 'spacy_output':[[0.12,0.14,3.5],[1.21,0.84,1.92],[0.34,0.85,2.43],[0.09,0.18,2.21]], 'response':[0,1,1,0]}
df = pd.DataFrame(d)

# 处理嵌套的向量化列
spacy_features = pd.DataFrame(df['spacy_output'].tolist(), columns=[f'spacy_{i}' for i in range(len(df['spacy_output'][0]))])
df_processed = pd.concat([df.drop('spacy_output', axis=1), spacy_features], axis=1)

# 训练模型
X = df_processed.drop('response', axis=1).values
y = df_processed['response'].values.ravel()

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=23)

clf = RandomForestClassifier(min_samples_split=4, n_estimators=100, criterion='entropy')
clf.fit(X_train, y_train)

# 输出预测结果
y_pred = clf.predict(X_test)
print("预测结果：", y_pred)

内容的提问来源于stack exchange，提问作者chicagobeast12