使用Pathos多进程训练Keras模型调用result.get()出错

Hey there! Let’s work through this serialization issue you’re facing when calling result.get() with Pathos. The error trace points to pickle/dill struggling to serialize something passed to your training function—here are actionable steps to resolve this:

1. Move Pool Initialization Inside the Main Guard Block #

On Windows, multiprocessing requires pool creation to happen within the if __name__ == "__main__": block to avoid unintended process spawning. Your original code initializes the pool outside this block, which can cause serialization quirks. Adjust your code like this:

from pathos.multiprocessing import ProcessingPool as Pool
from pathos.helpers import freeze_support
import dill
import training  # Ensure your training module is properly imported

if __name__ == "__main__":
    freeze_support()
    # Initialize the pool inside the main guard block
    p = Pool()
    result = p.amap(
        training.train_critic, 
        models_actor_t, 
        models_critic, 
        models_critic_t, 
        batch_states_list, 
        batch_actions_list, 
        batch_rewards_list, 
        batch_new_states_list, 
        batch_terminals_list, 
        sub_epochs_list, 
        id_list
    )
    print(result)
    print(result.get())

2. Verify Keras Model Serialization with Dill #

Even with Pathos, some Keras components (like custom layers, lambda layers, or unregistered objects) might trip up dill. Test if a single model can be serialized/deserialized first:

import dill

# Test serialization of one model from your list
test_model = models_actor_t[0]
try:
    with open("test_model.pkl", "wb") as f:
        dill.dump(test_model, f)
    # Try loading it back to confirm
    with open("test_model.pkl", "rb") as f:
        loaded_model = dill.load(f)
    print("Model serialized successfully!")
except Exception as e:
    print(f"Serialization failed: {str(e)}")

If this fails, fix the model’s serializability:

• Register custom layers with keras.utils.get_custom_objects().update({"YourLayer": YourLayer})
• Replace lambda layers with explicitly defined custom layers
• Remove non-serializable attributes (like file handles) from your model

3. Test with Synchronous map First #

Asynchronous amap can obscure error details. Switch to synchronous map with a small subset of your models to get clearer feedback:

# Use a small test set (e.g., first 2 entries from each list)
result = p.map(
    training.train_critic, 
    models_actor_t[:2], 
    models_critic[:2], 
    models_critic_t[:2], 
    batch_states_list[:2], 
    batch_actions_list[:2], 
    batch_rewards_list[:2], 
    batch_new_states_list[:2], 
    batch_terminals_list[:2], 
    sub_epochs_list[:2], 
    id_list[:2]
)
print(result)

4. Isolate Dependencies in Your Training Function #

Ensure training.train_critic doesn’t rely on global variables or non-serializable objects. Pass all necessary dependencies as parameters instead of referencing them from the global scope—this eliminates hidden serialization issues.

内容的提问来源于stack exchange，提问作者Lennart S.