使用multiprocessing调用外部脚本时触发Too many open files错误

问题背景 #

我通过multiprocessing.Pool批量处理多个条目，每个条目都会调用subprocess执行外部Python脚本。处理前会创建两个临时文件作为脚本参数，脚本内部调用C扩展处理条目，之后解析输出JSON文件返回结果，临时文件会在finally块中删除。但处理3880-3920个条目时，触发了OSError: [Errno 24] Too many open files错误。

已完成的排查动作 #

• 确认临时文件能正常创建并被删除
• 已将ulimit的软限制和硬限制都设置为1000000
• 代码中所有文件均通过with语句打开，避免文件句柄泄漏
• 改用dask集群执行相同逻辑无此问题
• 通过lsof排查发现：每个条目对应的临时文件虽已删除，但文件描述符仍在各进程中累积

临时解决方案（小规模测试有效，待大规模验证） #

手动关闭mkstemp返回的文件描述符：

fd, name = tempfile.mkstemp()
try:
    ...
finally:
    os.close(fd)
    os.unlink(name)

原出错代码 #

def func(item, protein, ncpu):

    output = None

    item_id = item.id

    output_fname = tempfile.mkstemp(suffix='_output.json', text=True)[1]
    input_fname = tempfile.mkstemp(suffix='_input.pdbqt', text=True)[1]   # <-- 错误触发位置

    try:
        with open(input_fname, 'wt') as f:
            f.write(preprocess(item))   # 将item转为文本格式，逻辑无关

        python_exec = sys.executable
        cmd = f'{python_exec} script.py -i {input_fname} -p {protein} -o {output_fname} -c {ncpu}'
        subprocess.run(cmd, shell=True)

        with open(output_fname) as f:
            res = f.read()
            if res:
                res = json.loads(res)
                output = {'score': res['score'],
                          'block': res['poses']}

    finally:
        os.unlink(input_fname)
        os.unlink(output_fname)

    return item_id, output


with Pool(ncpu) as pool:
    for item_id, res in pool.imap_unordered(partial(func, **kwargs), tuple(items), chunksize=1):
        yield item_id, res

错误信息 #

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/pavlop/anaconda3/envs/vina_cache/lib/python3.9/multiprocessing/pool.py", line 125, in worker
  File "/home/pavlop/python/docking-scripts/moldock/vina_dock.py", line 93, in func
OSError: [Errno 24] Too many open files: '/var/tmp/pbs.147815.login/tmpp8tqblfv_input.pdbqt'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/pavlop/python/docking-scripts/moldock/run_dock.py", line 230, in <module>
    main()
  File "/home/pavlop/python/docking-scripts/moldock/run_dock.py", line 203, in main
    for i, (item_id, res) in enumerate(docking(mols,
  File "/home/pavlop/python/docking-scripts/moldock/run_dock.py", line 74, in docking
    for item_id, res in pool.imap_unordered(partial(func, **kwargs), tuple(items), chunksize=1):
  File "/home/pavlop/anaconda3/envs/vina_cache/lib/python3.9/multiprocessing/pool.py", line 870, in next
    raise value
OSError: [Errno 24] Too many open files: '/var/tmp/pbs.147815.login/tmpp8tqblfv_input.pdbqt'

疑问 #

• 代码中哪里操作有误？
• 为何文件描述符没有自动释放？
• 是否可能是C扩展未释放文件描述符导致的问题？

内容的提问来源于stack exchange，提问作者DrDom