Imblearn Pipeline+GridSearchCV训练后模型保存遇pickle锁错误求助

I’ve run into this exact pickle issue before when combining Keras models with imbalanced-learn pipelines and scikit-learn’s GridSearchCV. Let’s break down why this happens and how to fix it:

Why This Error Occurs #

The _thread.RLock pickle error pops up because:

• The full GridSearchCV object (grid_result) contains lots of transient, non-serializable components—like cross-validation state, thread pool handles from n_jobs=-1, and internal locks in Keras/SMOTE.
• Even the best_estimator_ (your pipeline) has nested objects that pickle can’t handle cleanly, especially the KerasClassifier wrapper which holds onto internal Keras session locks.

Step-by-Step Solution: Split and Save Components Separately #

Instead of saving the entire pipeline or GridSearchCV object, split out the individual components that you need, and save each with the appropriate method:

1. Save the Preprocessing & Oversampling Components (Scaler + SMOTE)

Use joblib for these scikit-learn/imblearn objects—they serialize cleanly:

import joblib

# Extract components from the best pipeline
best_pipeline = grid_result.best_estimator_

# Save scaler
joblib.dump(best_pipeline.named_steps['standardize'], 'scaler.pkl')

# Save SMOTE
joblib.dump(best_pipeline.named_steps['oversample'], 'smote.pkl')

2. Save the Keras Model Separately

The KerasClassifier wrapper holds the actual Keras model in its .model attribute. Use Keras’s native save method instead of pickle/joblib for this:

# Save the underlying Keras model (not the wrapper)
best_pipeline.named_steps['NN_clf'].model.save('best_keras_model.h5')

3. Loading the Model Later

To reconstruct your full pipeline for inference:

import joblib
import tensorflow as tf
from sklearn.pipeline import Pipeline
from imblearn.over_sampling import SMOTE
from sklearn.preprocessing import StandardScaler
from tensorflow.keras.wrappers.scikit_learn import KerasClassifier

# Load preprocessing components
scaler = joblib.load('scaler.pkl')
smote = joblib.load('smote.pkl')

# Load Keras model
keras_model = tf.keras.models.load_model('best_keras_model.h5')

# Re-wrap the Keras model with KerasClassifier (reuse your original create_model function)
def create_model():
    # We only need the structure to match; the loaded weights will override it
    model = tf.keras.models.Sequential([
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Input(shape=(D,)),
        # ... match your original layer structure exactly ...
        tf.keras.layers.Dense(units=1,activation='sigmoid')
    ])
    model.compile(optimizer='Adam', loss='binary_crossentropy', metrics=['binary_accuracy'])
    return model

# Wrap the loaded model
nn_clf = KerasClassifier(build_fn=create_model)
nn_clf.model = keras_model

# Reconstruct the pipeline
reconstructed_pipeline = Pipeline([
    ('standardize', scaler),
    ('oversample', smote),
    ('NN_clf', nn_clf)
])

# Now you can use reconstructed_pipeline.predict(X_test) as usual

Quick Side Note: Fix Your Parameter Grid #

I noticed a small issue in your nn_param_grid that might cause training problems (even though it’s not the source of the pickle error):

• You’re passing 'L1L2' as a string for regularization_fn, but your create_model expects a regularizer instance. Update that line to:

  'NN_clf__regularization_fn': [tf.keras.regularizers.l1_l2(0.001)], # Pass actual regularizer instances, not strings
  

This approach avoids trying to serialize non-pickleable lock objects by separating the components that play nice with joblib/pickle from the Keras model, which has its own robust save system.

内容的提问来源于stack exchange，提问作者Surajit Chakraborty