如何在Python multiprocessing Pool中为函数传递多个参数？

我写了一个网页爬取编译数据的程序，原本只传单个参数就能调用函数完成操作，示例代码如下：

import multiprocessing

Links = [
        "https://www.[example].com",
        "https://www.[example2].com"
        
]
def extract_and_compile(link:str, compile_amount:int = 5):
    # 示例代码，非真实逻辑
    for x in range(compile_amount):
        print(f"compiled data {x+1} from {link}")
    

if __name__ == "__main__":
    with multiprocessing.Pool() as pool:
        pool.map(extract_and_compile, Links)

这段代码运行正常，输出符合预期：

compiled data 1 from https://www.[example].com
...
compiled data 5 from https://www.[example].com
compiled data 1 from https://www.[example2].com
...
compiled data 5 from https://www.[example2].com

现在想通过multiprocessing.Pool修改compile_amount参数，但直接传(Links,2)给pool.map会得到非预期输出：

if __name__ == "__main__":
    with multiprocessing.Pool() as pool:
        pool.map(extract_and_compile, (Links,2))

输出结果：

compiled data 1 from 2
...
compiled data 1 from ['https://www.[example].com', 'https://www.[example2].com']
...

请问怎么实现通过Pool给函数传递第二个参数，得到正确的执行结果？

方法1：用functools.partial固定参数 #

functools.partial可以预先绑定函数的指定参数，生成一个适配pool.map的单参数函数：

import multiprocessing
from functools import partial

Links = [
        "https://www.[example].com",
        "https://www.[example2].com"
]
def extract_and_compile(link:str, compile_amount:int = 5):
    for x in range(compile_amount):
        print(f"compiled data {x+1} from {link}")
    

if __name__ == "__main__":
    # 固定compile_amount为2
    target_func = partial(extract_and_compile, compile_amount=2)
    with multiprocessing.Pool() as pool:
        pool.map(target_func, Links)

方法2：用pool.starmap传递多参数元组 #

starmap支持接收元组列表，每个元组对应函数的一组参数，会自动解包传递：

import multiprocessing

Links = [
        "https://www.[example].com",
        "https://www.[example2].com"
]
def extract_and_compile(link:str, compile_amount:int = 5):
    for x in range(compile_amount):
        print(f"compiled data {x+1} from {link}")
    

if __name__ == "__main__":
    # 把每个link和compile_amount打包成元组
    args_list = [(link, 2) for link in Links]
    with multiprocessing.Pool() as pool:
        pool.starmap(extract_and_compile, args_list)

方法3：修改函数接收元组参数（不推荐） #

直接调整函数参数格式，让它接收元组并手动拆分：

import multiprocessing

Links = [
        "https://www.[example].com",
        "https://www.[example2].com"
]
# 修改函数参数为元组
def extract_and_compile(args):
    link, compile_amount = args
    compile_amount = compile_amount or 5  # 保留默认值逻辑
    for x in range(compile_amount):
        print(f"compiled data {x+1} from {link}")
    

if __name__ == "__main__":
    args_list = [(link, 2) for link in Links]
    with multiprocessing.Pool() as pool:
        pool.map(extract_and_compile, args_list)

优先推荐前两种方法，尤其是functools.partial，无需修改原函数逻辑，代码更简洁易维护。

内容的提问来源于stack exchange，提问作者Foxym