Python多进程场景下如何基于共享内存实现支持读写的共享字典?
Python多进程共享字典高性能解决方案
方案1:基于Copy-On-Write+本地缓存的标准库实现(无额外依赖,最适配读多写少场景)
你的场景读操作远多于写、总数据量不超过10MB,完全可以利用Linux系统下进程fork的Copy-On-Write机制实现接近本地字典的读取性能,不需要引入任何第三方依赖:
- 父进程维护全局主字典,子进程fork启动时会直接共享父进程的主字典内存页,只要不修改就不会产生内存复制开销
- 每个子进程维护本地待写入缓存,读取时优先查本地继承的主字典,再查本地缓存,都未命中再评估新的候选解
- 子进程的待写入缓存攒到指定条数或定期通过
multiprocessing.Queue批量同步给父进程,父进程统一更新主字典后再把增量推送给所有子进程更新本地副本
示例代码如下:
import multiprocessing as mp import os import time def worker(a, b, c, input_queue, output_queue): # 初始本地字典,fork后共享父进程内存,无复制开销 local_dict = {} local_cache = {} cache_threshold = 100 while True: # 先处理父进程发来的增量更新 while not input_queue.empty(): update = input_queue.get() local_dict.update(update) # 你的业务逻辑 # 读操作:优先查local_dict,再查local_cache # candidate = ... # if candidate in local_dict or candidate in local_cache: # continue # 评估候选解逻辑 # result = ... # local_cache[candidate] = result # 缓存达到阈值,批量提交给父进程 if len(local_cache) >= cache_threshold: output_queue.put(local_cache) local_cache.clear() # 你的退出逻辑 # if 满足退出条件: # break # 退出前提交剩余缓存 if local_cache: output_queue.put(local_cache) return worker_result def main(): a, b, c = 1, 2, 3 n_processes = os.cpu_count() main_dict = {} # 每个子进程对应一个输入队列,用于接收增量更新 input_queues = [mp.Queue() for _ in range(n_processes)] # 统一输出队列,子进程提交待写入数据 output_queue = mp.Queue() with mp.Pool(processes=n_processes) as pool: async_results = [pool.apply_async(worker, (a, b, c, input_queues[i], output_queue)) for i in range(n_processes)] # 主进程负责同步数据 while any(not res.ready() for res in async_results): # 收集子进程提交的更新 updates = {} while not output_queue.empty(): updates.update(output_queue.get()) if updates: main_dict.update(updates) # 把增量推给所有子进程 for q in input_queues: q.put(updates) time.sleep(0.1) results = [res.get() for res in async_results] # 处理最终结果 if __name__ == '__main__': main()
方案2:使用现成共享内存字典库(性能最优,改动量小)
如果你不想修改现有业务逻辑,可以直接用基于Python标准库multiprocessing.shared_memory实现的第三方库shared-memory-dict,内部已经封装好了并发控制、哈希表实现,不需要你从零开发,性能比Manager.dict高10~100倍,完全支持你需要的frozenset键、列表/集合值的存储:
用法示例:
# 先安装:pip install shared-memory-dict from shared_memory_dict import SharedMemoryDict import multiprocessing as mp import os def worker(a, b, c, smd_name): shared_dict = SharedMemoryDict(name=smd_name, size=10*1024*1024) # 10MB大小,和你预估的容量匹配 while condition: # 业务逻辑不变,直接读写shared_dict即可 ... return worker_result def main(): a, b, c = 1, 2, 3 # 初始化共享内存字典,设置10MB容量 shared_dict = SharedMemoryDict(name='heuristic-cache', size=10*1024*1024) n_processes = os.cpu_count() with mp.Pool(processes=n_processes) as pool: async_results = [pool.apply_async(worker, (a, b, c, 'heuristic-cache')) for _ in range(n_processes)] results = [res.get() for res in async_results] # 用完释放共享内存 shared_dict.shm.close() shared_dict.shm.unlink() if __name__ == '__main__': main()
方案3:现有Manager.dict优化(改动最小,快速见效)
如果你不想换架构也不想装第三方库,只需要给现有逻辑加一层本地缓存,就能大幅提升读取性能:
import multiprocessing as mp import os from functools import lru_cache def worker(a, b, c, shared_dict): # 加本地缓存,避免重复走管道读共享字典 @lru_cache(maxsize=None) def get_cached(key): return shared_dict.get(key) while condition: # 读的时候调用get_cached,写的时候同时更新缓存和共享字典 # candidate = ... # if get_cached(candidate) is not None: # continue # 评估逻辑 # result = ... # shared_dict[candidate] = result # get_cached.cache_clear() # 写入后清除缓存,避免脏读,或者也可以手动更新缓存 return worker_result def main(): with mp.Manager() as manager: shared_dict = manager.dict() n_processes = os.cpu_count() with mp.Pool(processes=n_processes) as pool: async_results = [pool.apply_async(worker, (1,2,3, shared_dict)) for _ in range(n_processes)] results = [res.get() for res in async_results] if __name__ == '__main__': main()
内容的提问来源于stack exchange,提问作者Sirion
相关产品推荐
相关产品推荐

