H2O AutoML训练混淆矩阵仅显示10k样本而非200k的原因排查

使用H2O AutoML训练二分类模型，原始数据集约300k行，按时间敏感拆分得到：训练集（70%，约200k行）、验证集（10%，约30k行）、测试集（10%，约30k行）、混合集（10%，约30k行）。但查看训练集混淆矩阵时，仅显示约10k样本，而非预期的200k行；验证集和测试集的混淆矩阵则正常显示约30k样本。

模型创建代码 #

#  Create model
aml = H2OAutoML(
    max_runtime_secs=max_runtime_secs,
    stopping_metric=stopping_metric, # "AUCPR"
    sort_metric=sort_metric, # "AUCPR"
    nfolds=nfolds, # set to 0
    distribution=distribution, # "bernoulli"
    verbosity=verbosity,
    balance_classes=balance_classes, # False
    seed=seed,
)

aml.train(
    y=outcome_column,
    training_frame=train,
    validation_frame=valid,
    leaderboard_frame=test,
    blending_frame=blend,
)

# Get the best model
best_model = aml.get_best_model()

# get the performance on test
performance = best_model.model_performance(test)

# define the threshold based on the desired metric
best_threshold = best_model.find_threshold_by_max_metric(
        metric=metric_to_use, valid=True)

# inspect confusion matrix on training set using that threshold
train_confusion = best_model.confusion_matrix(
        thresholds=best_threshold, train=True)

# inspect confusion matrix on test using that threshold
test_confusion = performance.confusion_matrix(thresholds=best_threshold)

# confusion matrix validation using that threshold
valid_confusion = best_model.confusion_matrix(
    thresholds=best_threshold, valid=True)

混淆矩阵结果 #

confusion matrix train: Confusion Matrix (Act/Pred) @ threshold = 0.35701837501784456
       False    True    Error    Rate
-----  -------  ------  -------  --------------
False  8589     190     0.0216   (190.0/8779.0)
True   272      904     0.2313   (272.0/1176.0)
Total  8861     1094    0.0464   (462.0/9955.0) 

confusion matrix valid: Confusion Matrix (Act/Pred) @ threshold = 0.3555305434918455
       False    True    Error    Rate
-----  -------  ------  -------  ----------------
False  23367    802     0.0332   (802.0/24169.0)
True   1486     1580    0.4847   (1486.0/3066.0)
Total  24853    2382    0.084    (2288.0/27235.0) 

confusion matrix test: Confusion Matrix (Act/Pred) @ threshold = 0.3546996890950105
       False    True    Error    Rate
-----  -------  ------  -------  ----------------
False  23399    769     0.0318   (769.0/24168.0)
True   1537     1529    0.5013   (1537.0/3066.0)
Total  24936    2298    0.0847   (2306.0/27234.0) 

模型排行榜 #

LEADERBOARD:
model_id                                                    aucpr       auc    logloss    mean_per_class_error      rmse        mse    training_time_ms    predict_time_per_row_ms  algo
StackedEnsemble_BestOfFamily_1_AutoML_1_20230504_164001  0.632635  0.876718   0.226965                0.260764  0.253097  0.0640579                4397                   0.041607  StackedEnsemble
GBM_1_AutoML_1_20230504_164001                           0.632514  0.876067   0.237024                0.262188  0.254668  0.0648556               24116                   0.038866  GBM
StackedEnsemble_BestOfFamily_2_AutoML_1_20230504_164001  0.631139  0.87681    0.22705                 0.262454  0.253345  0.0641838                1421                   0.049998  StackedEnsemble
GBM_4_AutoML_1_20230504_164001                           0.491181  0.810158   0.338646                0.308554  0.311599  0.0970939                 638                   0.001294  GBM
GBM_3_AutoML_1_20230504_164001                           0.374094  0.762457   0.342545                0.353879  0.312943  0.0979334                 642                   0.001166  GBM
DRF_1_AutoML_1_20230504_164001                           0.373471  0.755862   2.01974                 0.290401  0.311465  0.0970105                1735                   0.001758  DRF
GBM_2_AutoML_1_20230504_164001                           0.355587  0.758635   0.343282                0.371797  0.3132    0.0980943                 960                   0.001194  GBM
GLM_1_AutoML_1_20230504_164001                           0.330708  0.727998   0.312086                0.362253  0.298647  0.08919                 23648                   0.001599  GLM
[8 rows x 10 columns]

核心原因：StackedEnsemble的训练采样机制 #

从排行榜可见，最优模型是StackedEnsemble（堆叠集成模型）。H2O的StackedEnsemble在训练时，默认不会使用完整训练集生成混淆矩阵：

• 当基模型为树模型（如GBM、DRF）时，StackedEnsemble会采用基模型的袋外(OOB)样本或子采样样本来构建集成逻辑，这部分样本量远小于原始训练集。
• 你看到的10k样本正是该集成模型用于训练的采样子集，而非原始200k训练集。

验证方式 #

可以通过打印模型信息确认类型：

print(best_model)

输出会明确标注模型为StackedEnsemble，同时可查看其使用的基模型及采样策略细节。

解决方法 #

方法1：使用基模型计算完整训练集混淆矩阵 #

选择排行榜中表现接近的基模型（如GBM_1_AutoML_1_20230504_164001），直接计算完整训练集的混淆矩阵：

# 获取指定基模型
gbm_model = aml.get_model("GBM_1_AutoML_1_20230504_164001")
# 计算完整训练集混淆矩阵
full_train_confusion = gbm_model.confusion_matrix(thresholds=best_threshold, train=True)
print(full_train_confusion)

方法2：手动对完整训练集预测后统计混淆矩阵 #

用最优模型直接对完整训练集做预测，再手动统计混淆矩阵：

# 对完整训练集预测
train_preds = best_model.predict(train)
# 合并真实标签与预测结果
train_results = train[outcome_column].cbind(train_preds)
# 按指定阈值计算混淆矩阵
threshold = 0.35701837501784456
true_neg = ((train_results[outcome_column] == 0) & (train_results['p1'] < threshold)).sum()
false_pos = ((train_results[outcome_column] == 0) & (train_results['p1'] >= threshold)).sum()
false_neg = ((train_results[outcome_column] == 1) & (train_results['p1'] < threshold)).sum()
true_pos = ((train_results[outcome_column] == 1) & (train_results['p1'] >= threshold)).sum()

# 输出混淆矩阵
print(f"Confusion Matrix (Act/Pred) @ threshold = {threshold}")
print(f"       False    True")
print(f"-----  -------  ------")
print(f"False  {true_neg}     {false_pos}")
print(f"True   {false_neg}      {true_pos}")
print(f"Total  {true_neg+false_neg}     {false_pos+true_pos}")

方法3：强制StackedEnsemble保留完整训练集预测（不推荐） #

通过设置keep_training_predictions=True强制模型保留完整训练集的预测结果，但会大幅增加内存消耗，不适用于超大数据集：

aml = H2OAutoML(
    max_runtime_secs=max_runtime_secs,
    stopping_metric=stopping_metric,
    sort_metric=sort_metric,
    nfolds=nfolds,
    distribution=distribution,
    verbosity=verbosity,
    balance_classes=balance_classes,
    seed=seed,
    keep_training_predictions=True  # 新增参数
)

内容的提问来源于stack exchange，提问作者Guest6117