使用pickle序列化对象遇_thread.lock无法序列化问题咨询

Let's tackle this problem head-on—first, we'll break down why the error happens, fix your test code, then walk through how to track down the hidden lock in your main project.

Why This Error Occurs #

Pickle can't serialize low-level synchronization primitives like _thread.lock (the underlying object wrapped by threading.Lock) because these are tightly tied to your OS's thread resources. They aren't designed to be saved and reloaded across different process contexts. In your test code, you're explicitly storing a threading.Lock as an instance variable (self.lock), which triggers the error when pickle tries to process it.

Fixing Your Test Code #

The cleanest approach is to tell pickle to exclude the lock during serialization, then reinitialize it when deserializing. We'll use pickle's built-in hooks: __getstate__ and __setstate__.

Modified Class with Pickle Hooks #

import threading
from time import sleep
import pickle

class some_class:
    def __init__(self):
        self.a = 1
        self.running = True  # Add a stop flag for clean thread shutdown
        thr = threading.Thread(target=self.incr)
        self.lock = threading.Lock()
        thr.start()
    
    def incr(self):
        while self.running:
            with self.lock:
                self.a += 1
                print(self.a)
                sleep(0.5)
    
    def stop(self):
        # Safely stop the thread before pickling to avoid orphaned threads
        self.running = False
    
    def __getstate__(self):
        # Return a copy of the instance state, excluding the lock
        state = self.__dict__.copy()
        del state['lock']
        return state
    
    def __setstate__(self, state):
        # Restore the state and reinitialize the lock
        self.__dict__.update(state)
        self.lock = threading.Lock()
        # Optional: Restart the thread if needed for your use case
        self.running = True
        thr = threading.Thread(target=self.incr)
        thr.start()

if __name__ == "__main__":
    a = some_class()
    sleep(2)  # Let the thread run briefly
    a.stop()  # Stop the thread before pickling
    val = pickle.dumps(a, pickle.HIGHEST_PROTOCOL)
    print("pickle done!")
    
    # Test deserialization
    restored_a = pickle.loads(val)
    print(f"Restored value of a: {restored_a.a}")

This solution:

• Removes the lock from the data pickle serializes
• Re-creates a fresh lock when the object is loaded
• Adds a stop flag to safely shut down the thread before serialization

Tracking Down Hidden Locks in Your Main Project #

Since you aren't using locks directly, the issue is likely coming from a third-party library that uses thread-safe internals (like connection pools, caches, or async utilities). Here's how to find and fix it:

1. Locate the Lock in Your Object Graph #

Write a helper function to recursively scan your object for locks:

import threading

def find_locks(obj, current_path=""):
    """Recursively search for threading.Lock instances in an object."""
    if hasattr(obj, '__dict__'):
        for attr_name, attr_value in obj.__dict__.items():
            path = f"{current_path}.{attr_name}" if current_path else attr_name
            if isinstance(attr_value, threading.Lock):
                print(f"Found Lock at: {path}")
            # Skip basic types to avoid unnecessary recursion
            elif not isinstance(attr_value, (int, str, float, list, dict, tuple, set)):
                find_locks(attr_value, path)

# Use it before pickling your main object
find_locks(your_main_object)

This will print the exact path to any locks hidden in your object's attributes.

2. Identify the Third-Party Source #

Once you know which attribute has the lock, trace back to which library created it. Common culprits include:

• Database connection pools (e.g., SQLAlchemy, psycopg2)
• Caching libraries (e.g., redis-py with connection pooling)
• Async frameworks (e.g., aiohttp, asyncio)
• Thread-safe data structures from libraries like concurrent.futures

3. Fix the Third-Party Object Issue #

Depending on the library, you have a few options:

• Use the library's serialization method: Some libraries provide built-in ways to serialize their objects (e.g., to_dict() or serialize() methods).
• Exclude the problematic attribute: Use __getstate__ in your own class to remove the library object containing the lock, then reinitialize it after deserialization.
• Switch to a pickle alternative: Libraries like dill extend pickle's capabilities and can serialize more complex objects (including some lock types). Install it with pip install dill, then replace pickle with dill:

  import dill
  val = dill.dumps(your_main_object, dill.HIGHEST_PROTOCOL)
  

  Note: Dill isn't always compatible with all objects, so test thoroughly.

内容的提问来源于stack exchange，提问作者Masoud Rahimi