TensorFlow Keras 多进程推理触发_thread.RLock无法序列化报错
问题描述
尝试通过多进程机制加速Keras模型推理逻辑,需将模型传入各执行单元完成输入预测及后续计算,运行时抛出TypeError: can't pickle _thread.RLock objects错误,相关问题已在TensorFlow官方GitHub仓库提交反馈。
已尝试的无效解决方案
- 借助
partial固定模型参数生成偏函数,传入多进程池执行 - 为每个执行单元克隆独立的模型实例使用
- 为每个执行单元实现模型的保存-重载流程
- 替换为
pathos.multiprocessing实现多进程逻辑
最小可复现代码
import tensorflow as tf from tensorflow import keras import numpy as np from multiprocessing import Pool # from multiprocessing.dummy import Pool as ThreadPool # from pathos.multiprocessing import ProcessingPool as Pool from functools import partial def simple_model(): model = keras.models.Sequential([ keras.layers.Dense(units = 10, input_shape = [1]), keras.layers.Dense(units = 1, activation = 'sigmoid') ]) model.compile(optimizer = 'sgd', loss = 'mean_squared_error') return model def clone_model(model): model_clone = tf.keras.models.clone_model(model) model_clone.set_weights(model.get_weights()) model_clone.build((None, 1)) model_clone.compile(optimizer = 'sgd', loss = 'mean_squared_error') return model_clone def work(model, seq): return model.predict(seq) def load_model(model_savepath): return tf.keras.models.load_model(model_savepath) def worker(model, n = 4): seqences = np.arange(0,100).reshape(n, -1) pool = Pool() model_savepath = './simple_model.h5' model.save(model_savepath) model_list = [load_model(model_savepath) for _ in range(n)] # model_list = [clone_model(model) for _ in range(n)] results = pool.map(work, zip(model_list,seqences)) # partial_work = partial(work, model=model) # results = pool.map(partial_work, seqences) pool.close() pool.join() return np.reshape(results, (-1, )) if __name__ == '__main__': model = simple_model() out = worker(model, n=4) print(out)
错误堆栈
File "c:/Users/***/Documents/GitHub/COVID-NSF/test4.py", line 42, in <module> out = worker(model, n=4) File "c:/Users/****/Documents/GitHub/COVID-NSF/test4.py", line 30, in worker results = pool.map(work, zip(model_list,seqences)) File "C:\Users\****\anaconda3\envs\tf-gpu\lib\multiprocessing\pool.py", line 268, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "C:\Users\****\anaconda3\envs\tf-gpu\lib\multiprocessing\pool.py", line 657, in get raise self._value File "C:\Users\***\anaconda3\envs\tf-gpu\lib\multiprocessing\pool.py", line 431, in _handle_tasks put(task) File "C:\Users\***\anaconda3\envs\tf-gpu\lib\multiprocessing\connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "C:\Users\***\anaconda3\envs\tf-gpu\lib\multiprocessing\reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) TypeError: can't pickle _thread.RLock objects
问题根因
Windows环境下Python多进程默认采用spawn启动模式,主进程需要将所有传递给子进程的任务参数、可执行函数通过pickle序列化后,再发送给子进程反序列化执行。TensorFlow/Keras的模型对象内部持有_thread.RLock类型的线程锁,这类对象不支持pickle序列化,只要主进程尝试将模型对象作为任务参数传递给子进程,就会触发上述报错。
之前所有尝试无效的核心原因:完全没有绕开「主进程序列化模型对象」的环节——不管是克隆模型、在主进程预加载多个模型实例、还是用偏函数固定模型参数,本质上都是把主进程内存中的模型对象作为序列化数据的一部分发送给子进程,必然触发pickle失败。即使使用支持更宽松序列化规则的第三方多进程库,依然可能遇到线程锁对象无法跨进程传递的兼容性问题。
稳定可运行的实现方案
利用多进程池的初始化回调机制,在每个子进程启动阶段独立加载模型,全程不需要主进程传递模型对象给子进程,从根源上避开模型序列化环节。
修正后的可运行代码:
import tensorflow as tf from tensorflow import keras import numpy as np from multiprocessing import Pool # 子进程全局空间存储模型实例 _subprocess_model = None def simple_model(): model = keras.models.Sequential([ keras.layers.Dense(units=10, input_shape=[1]), keras.layers.Dense(units=1, activation='sigmoid') ]) model.compile(optimizer='sgd', loss='mean_squared_error') return model # 多进程池初始化回调:每个子进程启动时仅执行一次 def init_worker(model_savepath): global _subprocess_model # 子进程内部独立加载模型,无主进程序列化传递环节 _subprocess_model = tf.keras.models.load_model(model_savepath) def work(seq): # 直接复用子进程全局空间内的模型做推理 return _subprocess_model.predict(seq) def worker(model, n=4): seqences = np.arange(0, 100).reshape(n, -1) model_savepath = './simple_model.h5' model.save(model_savepath) # 启动进程池时传入初始化逻辑,每个子进程独立完成模型加载 pool = Pool(processes=n, initializer=init_worker, initargs=(model_savepath,)) results = pool.map(work, seqences) pool.close() pool.join() return np.reshape(results, (-1,)) if __name__ == '__main__': model = simple_model() out = worker(model, n=4) print(out)
实现特性
- 每个子进程仅在启动时加载一次模型,后续推理任务直接复用内存中的模型实例,无反复加载的性能损耗
- 子进程内的模型实例相互独立,不存在跨进程的资源竞争问题
- 方案完全基于Python标准库multiprocessing实现,不需要依赖第三方多进程兼容库,在Windows、Linux环境下均可稳定运行
内容的提问来源于stack exchange,提问作者kosa
相关产品推荐
相关产品推荐

