1. Is Your Understanding of Data Leakage Correct? #

Absolutely, your understanding is 100% accurate—this is a classic and high-impact data leakage scenario in cross-validation.

When you preprocess the entire training dataset first (filling missing values with global means/modes, applying one-hot encoding using global category counts) before splitting into folds, you’re leaking information from the "unseen" validation fold into the training process. For example, the mean used to fill missing values in the validation fold is calculated using data from that very fold, which makes your model’s validation performance artificially optimistic. In real-world deployment, your model won’t have access to future data’s statistical properties, so this approach leads to unreliable, overinflated evaluation metrics.

The correct workflow is to fit your preprocessing steps only on the training subset of each fold, then apply that fitted preprocessor to both the training subset and its corresponding validation subset. This mimics how the model would behave with truly unseen data.

2. Implementing Custom Imputation in sklearn.pipeline.Pipeline #

Yes, you’ll need to create a custom transformer by subclassing sklearn.base.BaseEstimator and sklearn.base.TransformerMixin (the latter gives you a default fit_transform method for free). Here’s a practical, tailored implementation:

Step 1: Build the Custom Imputer Class #

This class will calculate and store mean values for float64 columns, and mode values for all other column types, then apply those values during transformation.

import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin

class CustomImputer(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        # Store imputation values per column
        self.impute_values_ = {}
        for col in X.columns:
            if X[col].dtype == np.float64:
                # Use mean for float64 columns
                self.impute_values_[col] = X[col].mean()
            else:
                # Use mode for other columns (pick first mode if there are ties)
                self.impute_values_[col] = X[col].mode()[0]
        return self
    
    def transform(self, X):
        # Apply precomputed imputation values
        X_imputed = X.copy()
        for col, val in self.impute_values_.items():
            X_imputed[col] = X_imputed[col].fillna(val)
        return X_imputed

Step 2: Integrate with Pipeline & Cross-Validation #

Now you can plug this custom imputer into a Pipeline alongside other preprocessing steps (like one-hot encoding) and your model. When paired with k-fold cross-validation, the pipeline ensures preprocessing is fitted only on each fold’s training data—eliminating leakage:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, KFold

# Example: Split columns into float64 and non-float groups
float_cols = [col for col in X_train.columns if X_train[col].dtype == np.float64]
non_float_cols = [col for col in X_train.columns if col not in float_cols]

# Create preprocessor: one-hot encode non-float cols, keep float cols as-is
preprocessor = ColumnTransformer(
    transformers=[
        ('onehot', OneHotEncoder(sparse_output=False, handle_unknown='ignore'), non_float_cols)
    ],
    remainder='passthrough'
)

# Full pipeline: impute → preprocess → model
pipeline = Pipeline([
    ('imputer', CustomImputer()),
    ('preprocessor', preprocessor),
    ('model', LogisticRegression())
])

# Run 5-fold cross-validation with no leakage
kf = KFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(pipeline, X_train, y_train, cv=kf, scoring='accuracy')

print(f"Cross-Validation Accuracy Scores: {scores}")
print(f"Mean Accuracy: {scores.mean():.4f}")

Quick Tips: #

• The CustomImputer follows scikit-learn’s estimator API, so it works seamlessly with all sklearn pipeline and cross-validation tools.
• handle_unknown='ignore' in OneHotEncoder prevents errors if the validation fold has categories not seen in the training fold.
• Using ColumnTransformer lets you apply different preprocessing logic to different column groups cleanly.

内容的提问来源于stack exchange，提问作者anon_swe