PyPy环境下使用Python multiprocessing Pool出现OSError: [Errno 24] Too many open files错误的求助
PyPy环境下使用Python multiprocessing Pool出现OSError: [Errno 24] Too many open files错误的求助
Edit: I'm using
pypy.
我尝试使用multiprocessing模块创建一个包含4个工作进程的Pool来并行化我的Python脚本。这个错误似乎表明打开了太多管道,但我实在想不通为什么会这样——毕竟我只生成了4个工作进程。
以下是生成Pool的函数:
def do_learning_experiments(model, alphabet, correct_size, prot): """ Perform the learning experiments for the given model and alphabet. Args: model: model to learn alphabet: input alphabet correct_size: correct size of the model """ # create a copy of the SUL for each oracle suls = [AutomatonSUL(model) for _ in range(NUM_ORACLES)] wpr = WALKS_PER_ROUND[prot] wl = WALK_LEN[prot] # initialize the oracles eq_oracles = [StochasticRandom(alphabet, suls[0], wpr, wl), StochasticLinear(alphabet, suls[1], wpr, wl), StochasticSquare(alphabet, suls[2], wpr, wl), StochasticExponential(alphabet, suls[3], wpr, wl)] # create the arguments for eache oracle's task tasks = [(alphabet, sul, oracle, correct_size, i) for i, (sul, oracle) in enumerate(zip(suls, eq_oracles))] with mp.Pool(processes=mp.cpu_count()) as pool: results = pool.starmap(process_oracle, tasks) return results
我在一个for循环中多次调用这个函数,目的是对测量结果取平均值,代码如下:
for trial in range(TIMES): results = do_learning_experiments(model, alphabet, correct_size, prot)
我注意到,当我设置TIMES=1或TIMES=10时,不会出现这个错误。但当我设置TIMES=30(这正是我需要的数值)时,错误就会出现。
这让我觉得进程没有被正确关闭,但在系统监视器里我任何时候都看不到超过5个Python进程在运行。
我该如何修复这个问题,或者至少进行调试呢?
备注:内容来源于stack exchange,提问作者Stefanos Anagnostou
相关产品推荐
相关产品推荐

