如何减少joblib并行运行时打开的文件数量？

我已在GitHub提交了issue，移除lsof中的user限制后，发现打开的文件数量更多。

简化复现案例（Python 3.11.8，joblib 1.4.0） #

案例1：普通函数 #

## lsof.py
from joblib import Parallel, delayed
import time
import sys

def f():
    time.sleep(5)
    return 1.0

def get_num_of_opened_files() -> int:
    from subprocess import run
    return int(run('lsof -u eastsun| wc -l', shell=True, capture_output=True, text=True).stdout.strip())


f0 = get_num_of_opened_files()
xs = Parallel(n_jobs=32, return_as='generator')(delayed(f)() for _ in range(32))
time.sleep(2)
f1 = get_num_of_opened_files()
print(f'Opened files: before {f0}, after {f1}, delta: {f1 - f0}', flush=True)
print(sum(xs))

运行上述脚本后显示，joblib运行时约打开了3200个文件：

>> python lsof.py
>> Opened files: before 388, after 3647, delta: 3259

案例2：带属性的实例方法 #

## lsof2.py
from joblib import Parallel, delayed
import time
import sys
import pandas as pd


class Tasker:
    def __init__(self):
        self.data = pd.Series([])

    def run(self):
        time.sleep(10)
        return 1.0

def get_num_of_opened_files() -> int:
    from subprocess import run
    return int(run('lsof -u eastsun| wc -l', shell=True, capture_output=True, text=True).stdout.strip())


tasker = Tasker()
f0 = get_num_of_opened_files()
xs = Parallel(n_jobs=32, return_as='generator')(delayed(tasker.run)() for _ in range(32))
time.sleep(2)
f1 = get_num_of_opened_files()
print(f'Opened files: before {f0}, after {f1}, delta: {f1 - f0}', flush=True)
print(sum(xs))

打开的文件数量翻倍：

>> python lsof2.py
>> Opened files: before 399, after 6794, delta: 6395

若运行10个此类程序，打开的文件数约达60000，且joblib会发出警告：

UserWarning: A worker stopped while some jobs were given to the
executor. This can be caused by a too short worker timeout or by a
memory leak.

核心疑问 #

如何减少joblib打开的文件数量？

我已确认以下几点：

• 若在Parallel中添加prefer='threads'，打开的文件数会降至约4个，但我需要的是多进程而非多线程。
• 将return_as参数从generator改为list无效果。
• 约60%的打开文件为.so文件，这让我十分困惑。

内容的提问来源于stack exchange，提问作者Eastsun