导入Numpy后初始化Python多进程Manager触发EOFError问题求助
导入Numpy后Python多进程Manager触发EOFError
之前运行Python多进程代码一切正常,仅添加了一行未实际调用的import numpy as np语句后,执行manager = mp.Manager()时就触发了EOFError。我了解Numpy和多进程存在一些历史问题,但从未遇到过这种仅导入就直接报错的情况,希望有人能解释原因。
相关代码如下:
import multiprocessing as mp import time import numpy as np fn = 'test_out.json' def worker(arg, q): '''stupidly simulates long running process''' start = time.perf_counter() s = 'this is a test' txt = s for i in range(200000): txt += s done = time.perf_counter() - start with open(fn, 'rb') as f: size = len(f.read()) res = 'Process' + str(arg), str(size), done q.put(res) return res def listener(q): '''listens for messages on the q, writes to file. ''' with open(fn, 'w') as f: while 1: m = q.get() if m == 'kill': f.write('killed') break f.write(str(m) + '\n') f.flush() def main(): #must use Manager queue here, or will not work manager = mp.Manager() q = manager.Queue() pool = mp.Pool(mp.cpu_count() + 2) #put listener to work first watcher = pool.apply_async(listener, (q,)) #fire off workers jobs = [] for i in range(80): job = pool.apply_async(worker, (i, q)) jobs.append(job) # collect results from the workers through the pool result queue for job in jobs: job.get() #now we are done, kill the listener q.put('kill') pool.close() pool.join() if __name__ == "__main__": main()
我原本以为添加未使用的Numpy导入不会影响代码运行结果。
内容的提问来源于stack exchange,提问作者JamesVR
相关产品推荐
相关产品推荐

