如何修改基于multiprocess模块的multi_proc_parallel_functions函数以支持传入可变长度参数列表（适配AWS Lambda环境）

Got it, let's adjust your code to support variable-length arguments for each function in the parallel process setup. Here's how to do it step by step:

1. Update the Worker Function (parallel_functions) #

We need this function to handle both the target function and its associated arguments. We'll use tuple unpacking to separate the function from its args:

def parallel_functions(func_args_tuple, send_end):
    # Unpack the function and its arguments from the tuple
    func, *args = func_args_tuple
    # Call the function with unpacked arguments and send the result
    send_end.send(func(*args))

2. Adjust the Multiprocess Coordinator (multi_proc_parallel_functions) #

The core logic here stays mostly the same, but we'll now iterate over a list of tuples (each tuple holds a function and its arguments) instead of just functions:

def multi_proc_parallel_functions(function_args_list, target_func):
    jobs = []
    pipe_list = []
    for func_args in function_args_list:
        recv_end, send_end = mp.Pipe(False)
        # Pass the entire (func, args...) tuple to the worker process
        p = mp.Process(target=target_func, args=(func_args, send_end))
        jobs.append(p)
        pipe_list.append(recv_end)
        p.start()
    # Collect results from all pipes
    result_list = [x.recv() for x in pipe_list]
    # Wait for all processes to finish
    for proc in jobs:
        proc.join()
    return result_list

3. Usage Example #

Now you can create a list of tuples where each tuple contains your function followed by its arguments. Here's how to call it with your updated adder functions:

import numpy as np
import multiprocess as mp

# Your updated adder functions
def adder10(x,y):
    return np.random.randint(5) + 10 + x * y

def adder1000(a,b, c):
    return np.random.randint(5) + 1000 -a + b +c

# Create a list of (function, arg1, arg2, ...) tuples
function_args_list = [
    (adder10, 2, 3),    # Calls adder10(2,3)
    (adder10, 1, 4),    # Calls adder10(1,4)
    (adder1000, 5, 2, 3), # Calls adder1000(5,2,3)
    (adder1000, 1, 1, 1)  # Calls adder1000(1,1,1)
]

# Execute the parallel calls
results = multi_proc_parallel_functions(function_args_list, parallel_functions)
print(results)
# Example output (varies due to randint): [19, 15, 1001, 1002]

Key Notes #

• Tuple Packing/Unpacking: By packing each function and its arguments into a tuple, we make it easy to pass everything needed for the function call to each worker process. The *args syntax in parallel_functions handles unpacking arguments of any length seamlessly.
• AWS Lambda Compatibility: This modification doesn't introduce any new modules—it just adjusts how data is passed between processes—so it should work perfectly with the multiprocess module you're already using in Lambda.

内容的提问来源于stack exchange，提问作者nipy