PyCaret 3.4.0与scikit-learn模型结果不一致及特征选择困惑

问题背景 #

我当前使用PyCaret 3.4.0（4.0版本缺少我业务场景所需的配置参数），尝试用Scikit-Learn复现其随机森林模型的训练结果。在PyCaret执行setup并获取转换后的数据后，分别用两种框架训练模型，但评估指标存在明显差异。同时，我计划用Sklearn的RFECV进行特征数量缩减，但缩减后的最优平均F1值处于PyCaret全特征模型和Sklearn全特征模型的结果之间，无法确定是否应采用该特征集合。

训练代码对比 #

PyCaret训练代码 #

compare_models(include=['rf'],cross_validation=True)

Scikit-Learn训练代码 #

scoring=['accuracy','precision','recall','f1','f1_macro','f1_weighted','f1_micro', 'roc_auc']
rfc = RandomForestClassifier(random_state=42, n_jobs=-1)
scores = cross_validate(rfc, xtrain_trans, ytrain_trans, scoring=scoring, cv=cv)

评估结果差异 #

PyCaret输出结果 #

Model   Accuracy    AUC Recall  Prec.   F1  Kappa   MCC TT (Sec)
rf  Random Forest Classifier    0.7164  0.7617  0.7164  0.7254  0.7137  0.4329  0.4416  0.25

Scikit-Learn输出结果 #

• Accuracy=0.6948717948717947
• AUC=0.7411665257819103
• Recall=0.6908791208791208
• Precision=0.7005056185644422
• F1=0.6867142420587947
• F1_macro=0.6910345427878428
• F1_weighted=0.6911863570749788
• F1_micro=0.6948717948717947

已确认的一致性配置 #

二者使用完全相同的交叉验证拆分器：

cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=42)

PyCaret的setup配置详情：

setup(xtrain, 
            target = 'Group',
            session_id=42,
            test_data=xtest, #None default
            imputation_type=None, 
            remove_multicollinearity=True,
            multicollinearity_threshold=0.70,
            remove_outliers=True,
            transformation=True,
            transformation_method='quantile',
            normalize=True,
            feature_selection=False,
            fold_strategy=cv,
            use_gpu=True)

特征选择决策困境 #

我尝试用Sklearn的RFECV进行特征数量缩减，得到的最优平均F1值为0.70——该值低于PyCaret全特征模型的F1值（0.7137），但高于Scikit-Learn全特征模型的F1值（0.6867），因此无法确定是否应采用特征缩减后的特征集合。

可复现代码 #

from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.model_selection import cross_validate
from sklearn.ensemble import RandomForestClassifier
import pandas as pd
from pycaret.classification import setup, compare_models

cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=42)

from sklearn.datasets import make_classification
X, y = make_classification(n_samples=100, n_features=5, n_informative=2, n_classes=2, flip_y=0.2, random_state=42)
dataset = pd.DataFrame(X)
dataset.columns = ['X1', 'X2', 'X3', 'X4', 'X5']
dataset['y'] = y
setup0 = setup(dataset,target = 'y',session_id=42,train_size=0.7,imputation_type=None,remove_multicollinearity=True,multicollinearity_threshold=0.70,remove_outliers=True,transformation=True,transformation_method='quantile',normalize=True,feature_selection=False,fold_strategy=cv,use_gpu=True)

# CV average results from PyCaret
compare_models(include=['rf'],sort='F1',errors='raise',cross_validation=True)

xtrain_trans = setup0.get_config('X_train_transformed')
ytrain_trans = setup0.get_config('y_train_transformed')
scoring=['accuracy', 'roc_auc', 'recall', 'precision', 'f1','f1_macro','f1_weighted','f1_micro']
rfc = RandomForestClassifier(random_state=42, n_jobs=-1)
scores = cross_validate(rfc, xtrain_trans, ytrain_trans, scoring=scoring, cv=cv)
scores=pd.DataFrame(scores)
scores=scores.mean().round(4)

# CV average results from sklearn
pd.DataFrame(scores).transpose()

内容的提问来源于stack exchange，提问作者Guillermo Paz