MNIST任务中验证集与测试集准确率差异过大求助

我需要完成不使用神经网络的MNIST手写数字识别任务，要求准确率≥0.7。训练随机森林模型后，验证集准确率达0.88，但测试集准确率仅0.2，这种巨大差异令我困惑。换用逻辑回归模型也出现相同情况。

原始代码 #

train = np.loadtxt('./data/digit/train.csv', delimiter=',', skiprows=1)
test = np.loadtxt('./data/digit/test.csv', delimiter=',', skiprows=1)
# creating variable for labels
train_label = train[:, 0]
# changing shape
train_img = np.resize(train[:, 1:], (train.shape[0], 28, 28))
test_img = np.resize(test, (test.shape[0], 28, 28))

#creating vectors
X_train = train_img.reshape(-1, 28 * 28).astype(np.float32)
X_test = test_img.reshape(-1, 28 * 28).astype(np.float32)
#and normalize them by the mean.
X_mean = X_train.mean(axis=0)
X__mean = X_test.mean(axis=0)
X_train -= X_mean
X_test -= X__mean 

#calculate covariation matrix and make svd for train
cov = np.dot(X_train.T, X_train) / X_train.shape[0]
U, S, V = np.linalg.svd(cov)
#and for test
test_cov = np.dot(X_test.T, X_test) / X_test.shape[0]
U_, S_, V_ = np.linalg.svd(test_cov)

#visualize and analise PCA
S_cumsum = np.cumsum(S) / np.sum(S)
px.line(S_cumsum)

#Change features array dimension
S_thr = 0.8  
n_comp = np.argmax(np.where(S_cumsum > S_thr, 1, 0))
X_train = np.dot(X_train, U[:, :n_comp])
X_test = np.dot(X_test, U_[:, :n_comp])

#divide x_train for train and validation
x_train, x_val, y_train, y_val,  = train_test_split(X_train, train_label, test_size=0.2, random_state=42)

#fit with RandomizedSearchCV
rf_params = {'criterion': ['gini', 'entropy'], 'n_estimators': range(50, 150, 10), 'max_depth': range(3,10), 'max_features': np.arange(0.3, 1.0, 0.1), 'min_samples_leaf': range(2, 5)}
rf_model = RandomizedSearchCV(RandomForestClassifier(), rf_params, n_iter = 15, cv = 7, random_state = 42)
rf_model.fit(x_train, y_train)

rf_model.best_score_ = 0.8750297619047619
Accuracy: 0.8751190476190476

pred = rf_model.predict(X_test) #... and 0.2 only

核心问题与修正方案 #

问题1：测试集归一化逻辑错误

你分别用训练集和测试集的均值做归一化，这会破坏数据分布的一致性。正确做法是仅用训练集的均值对测试集做归一化，避免引入数据泄露和分布偏移。

问题2：PCA变换不一致

你对训练集和测试集分别做SVD分解，并用各自的U矩阵降维。正确的PCA流程应该是用训练集得到的U矩阵统一处理训练集和测试集，确保两者在同一特征空间转换。

修正后的关键代码

# 归一化修正：用训练集均值处理测试集
X_mean = X_train.mean(axis=0)
X_train -= X_mean
X_test -= X_mean  # 不再使用测试集自身的均值

# PCA修正：仅用训练集的SVD结果处理测试集
cov = np.dot(X_train.T, X_train) / X_train.shape[0]
U, S, V = np.linalg.svd(cov)

# 可视化PCA累积方差
S_cumsum = np.cumsum(S) / np.sum(S)
px.line(S_cumsum)

# 降维：训练集和测试集共用训练集的U矩阵
S_thr = 0.8  
n_comp = np.argmax(np.where(S_cumsum > S_thr, 1, 0))
X_train = np.dot(X_train, U[:, :n_comp])
X_test = np.dot(X_test, U[:, :n_comp])  # 使用U而非U_

额外注意事项 #

• 确认测试集是否包含标签：MNIST官方test.csv通常无标签，若要评估测试集准确率，需使用官方提供的测试标签文件，或从训练集中拆分一部分作为测试集。
• 参数搜索可微调：修正核心问题后，可根据验证集表现调整随机森林的参数范围，进一步提升准确率。

内容的提问来源于stack exchange，提问作者Vladislav