You need to enable JavaScript to run this app. Python多进程场景下如何基于共享内存实现支持读写的共享字典?
优惠活动
大模型
产品
解决方案
定价
更多

Python多进程场景下如何基于共享内存实现支持读写的共享字典?

Python多进程共享字典高性能解决方案

方案1:基于Copy-On-Write+本地缓存的标准库实现(无额外依赖,最适配读多写少场景)

你的场景读操作远多于写、总数据量不超过10MB,完全可以利用Linux系统下进程fork的Copy-On-Write机制实现接近本地字典的读取性能,不需要引入任何第三方依赖:

  1. 父进程维护全局主字典,子进程fork启动时会直接共享父进程的主字典内存页,只要不修改就不会产生内存复制开销
  2. 每个子进程维护本地待写入缓存,读取时优先查本地继承的主字典,再查本地缓存,都未命中再评估新的候选解
  3. 子进程的待写入缓存攒到指定条数或定期通过multiprocessing.Queue批量同步给父进程,父进程统一更新主字典后再把增量推送给所有子进程更新本地副本
    示例代码如下:
import multiprocessing as mp
import os
import time

def worker(a, b, c, input_queue, output_queue):
    # 初始本地字典,fork后共享父进程内存,无复制开销
    local_dict = {}
    local_cache = {}
    cache_threshold = 100

    while True:
        # 先处理父进程发来的增量更新
        while not input_queue.empty():
            update = input_queue.get()
            local_dict.update(update)
        
        # 你的业务逻辑
        # 读操作:优先查local_dict,再查local_cache
        # candidate = ...
        # if candidate in local_dict or candidate in local_cache:
        #     continue
        # 评估候选解逻辑
        # result = ...
        # local_cache[candidate] = result

        # 缓存达到阈值,批量提交给父进程
        if len(local_cache) >= cache_threshold:
            output_queue.put(local_cache)
            local_cache.clear()
        
        # 你的退出逻辑
        # if 满足退出条件:
        #     break
    
    # 退出前提交剩余缓存
    if local_cache:
        output_queue.put(local_cache)
    return worker_result

def main():
    a, b, c = 1, 2, 3
    n_processes = os.cpu_count()
    main_dict = {}
    # 每个子进程对应一个输入队列,用于接收增量更新
    input_queues = [mp.Queue() for _ in range(n_processes)]
    # 统一输出队列,子进程提交待写入数据
    output_queue = mp.Queue()

    with mp.Pool(processes=n_processes) as pool:
        async_results = [pool.apply_async(worker, (a, b, c, input_queues[i], output_queue)) for i in range(n_processes)]
        
        # 主进程负责同步数据
        while any(not res.ready() for res in async_results):
            # 收集子进程提交的更新
            updates = {}
            while not output_queue.empty():
                updates.update(output_queue.get())
            if updates:
                main_dict.update(updates)
                # 把增量推给所有子进程
                for q in input_queues:
                    q.put(updates)
            time.sleep(0.1)
        
        results = [res.get() for res in async_results]
    
    # 处理最终结果

if __name__ == '__main__':
    main()

方案2:使用现成共享内存字典库(性能最优,改动量小)

如果你不想修改现有业务逻辑,可以直接用基于Python标准库multiprocessing.shared_memory实现的第三方库shared-memory-dict,内部已经封装好了并发控制、哈希表实现,不需要你从零开发,性能比Manager.dict高10~100倍,完全支持你需要的frozenset键、列表/集合值的存储:
用法示例:

# 先安装:pip install shared-memory-dict
from shared_memory_dict import SharedMemoryDict
import multiprocessing as mp
import os

def worker(a, b, c, smd_name):
    shared_dict = SharedMemoryDict(name=smd_name, size=10*1024*1024) # 10MB大小,和你预估的容量匹配
    while condition:
        # 业务逻辑不变,直接读写shared_dict即可
        ...
    return worker_result

def main():
    a, b, c = 1, 2, 3
    # 初始化共享内存字典,设置10MB容量
    shared_dict = SharedMemoryDict(name='heuristic-cache', size=10*1024*1024)
    n_processes = os.cpu_count()
    with mp.Pool(processes=n_processes) as pool:
        async_results = [pool.apply_async(worker, (a, b, c, 'heuristic-cache')) for _ in range(n_processes)]
        results = [res.get() for res in async_results]
    # 用完释放共享内存
    shared_dict.shm.close()
    shared_dict.shm.unlink()

if __name__ == '__main__':
    main()

方案3:现有Manager.dict优化(改动最小,快速见效)

如果你不想换架构也不想装第三方库,只需要给现有逻辑加一层本地缓存,就能大幅提升读取性能:

import multiprocessing as mp
import os
from functools import lru_cache

def worker(a, b, c, shared_dict):
    # 加本地缓存,避免重复走管道读共享字典
    @lru_cache(maxsize=None)
    def get_cached(key):
        return shared_dict.get(key)
    
    while condition:
        # 读的时候调用get_cached,写的时候同时更新缓存和共享字典
        # candidate = ...
        # if get_cached(candidate) is not None:
        #     continue
        # 评估逻辑
        # result = ...
        # shared_dict[candidate] = result
        # get_cached.cache_clear() # 写入后清除缓存,避免脏读,或者也可以手动更新缓存
    return worker_result

def main():
    with mp.Manager() as manager:
        shared_dict = manager.dict()
        n_processes = os.cpu_count()
        with mp.Pool(processes=n_processes) as pool:
            async_results = [pool.apply_async(worker, (1,2,3, shared_dict)) for _ in range(n_processes)]
            results = [res.get() for res in async_results]

if __name__ == '__main__':
    main()

内容的提问来源于stack exchange,提问作者Sirion

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.10.18 05:15:03