阿拉伯语文本多标签分类样本数不匹配错误排查与优化求助

错误原因 #

Chi2、mutual_info_classif等特征选择工具默认针对单标签任务，要求目标变量是一维数组。你传入的train_labels是(28332,24)的二维矩阵，工具会错误地将y的列数（24）识别为样本数量，导致与X的28332个样本数不匹配，触发Found input variables with inconsistent numbers of samples: [28332, 24]错误。

解决方案与代码优化 #

方案1：适配Binary Relevance框架（逐标签特征选择） #

Binary Relevance将多标签拆分为24个独立二分类任务，因此可为每个标签单独做特征选择后合并结果：

from sklearn.feature_selection import SelectKBest, chi2, mutual_info_classif
import numpy as np

# 加权融合Chi2与互信息得分（归一化后加权）
def weighted_fusion(chi_scores, mi_scores, chi_weight=0.5, mi_weight=0.5):
    chi_norm = (chi_scores - chi_scores.min()) / (chi_scores.max() - chi_scores.min())
    mi_norm = (mi_scores - mi_scores.min()) / (mi_scores.max() - mi_scores.min())
    return chi_weight * chi_norm + mi_weight * mi_norm

# 逐标签处理特征选择
selected_features_per_label = []
num_features_to_select = 5000  # 可根据任务调整

for label_idx in range(train_labels.shape[1]):
    # 取出当前标签的一维目标变量
    single_label = train_labels[:, label_idx]
    
    # 计算Chi2得分
    chi_selector = SelectKBest(chi2, k='all')
    chi_selector.fit(train_text_bow, single_label)
    chi_scores = chi_selector.scores_
    
    # 计算互信息得分
    mi_selector = SelectKBest(mutual_info_classif, k='all')
    mi_selector.fit(train_text_bow, single_label)
    mi_scores = mi_selector.scores_
    
    # 加权融合并选择Top K特征
    fused_scores = weighted_fusion(chi_scores, mi_scores)
    top_feature_indices = fused_scores.argsort()[::-1][:num_features_to_select]
    selected_features_per_label.append(top_feature_indices)

# 合并所有标签的特征（取并集，也可根据需求取交集）
all_selected_features = np.unique(np.concatenate(selected_features_per_label))
train_text_selected = train_text_bow[:, all_selected_features]

方案2：适配Label Powerset框架（转单标签后特征选择） #

Label Powerset将多标签组合转为唯一单标签，可先转换标签格式再做特征选择：

from sklearn.feature_selection import SelectKBest, chi2, mutual_info_classif
import numpy as np

# 将多标签矩阵转为单标签类别（每个标签组合对应唯一整数）
lp_labels = np.apply_along_axis(lambda x: tuple(x), 1, train_labels)
label_map = {label: idx for idx, label in enumerate(np.unique(lp_labels))}
lp_train_labels = np.array([label_map[label] for label in lp_labels])

# 计算双特征选择得分并融合
chi_selector = SelectKBest(chi2, k='all')
chi_selector.fit(train_text_bow, lp_train_labels)
chi_scores = chi_selector.scores_

mi_selector = SelectKBest(mutual_info_classif, k='all')
mi_selector.fit(train_text_bow, lp_train_labels)
mi_scores = mi_selector.scores_

# 加权融合并选择Top K特征
fused_scores = 0.5 * chi_scores + 0.5 * mi_scores
num_features_to_select = 5000
top_feature_indices = fused_scores.argsort()[::-1][:num_features_to_select]
train_text_selected = train_text_bow[:, top_feature_indices]

额外优化建议 #

• 先对BoW做TF-IDF转换：阿拉伯语文本高频无意义词多，用TfidfTransformer转换后再做特征选择，能提升得分区分度。
• 动态调整权重：根据任务需求调整Chi2与互信息的权重，比如侧重词频相关性时提高Chi2权重，侧重依赖关系时提高互信息权重。
• 交叉验证选特征数：通过交叉验证确定最优num_features_to_select，避免过拟合或特征不足。

内容的提问来源于stack exchange，提问作者hussain_ali_r