如何对内置Python multiprocessing的函数实现多进程批量调用？

Hey there! Let's tackle this problem step by step. First, a critical heads-up: since your existing complex function already uses multiprocessing, we need to avoid nested multiprocessing (this can cause resource leaks, crashes, or weird behavior, especially on Windows). With that in mind, here are two reliable approaches to process your variable range (10 to 100) using multiprocessing:

Approach 1: Use multiprocessing.Pool (Recommended for Simplicity) #

The Pool class handles process management automatically, so you don't have to manually create and track each process. It's perfect for batch processing tasks like yours.

import multiprocessing
from functools import partial

# This is the complex function you downloaded (already uses multiprocessing internally)
def complex_function(variable, extra_param=None):
    # Original multiprocessing logic lives here...
    # Simplified example output
    result = variable * 2
    return result

if __name__ == "__main__":
    # Define your variable range: 10 to 100 (inclusive)
    variables = range(10, 101)
    
    # If your complex function needs additional fixed parameters, use partial to wrap it
    # wrapped_func = partial(complex_function, extra_param="your_fixed_value")
    
    # Create a process pool - using cpu_count() balances performance and system load
    with multiprocessing.Pool(processes=multiprocessing.cpu_count()) as pool:
        # Use map() to process variables in order, returns results in the same order
        results = pool.map(complex_function, variables)
        
        # Alternatively, use imap_unordered() to get results as they finish (better for memory if results are large)
        # results = list(pool.imap_unordered(complex_function, variables))
    
    # Print or process the results
    for var, res in zip(variables, results):
        print(f"Variable {var} result: {res}")

Key Notes for This Approach: #

• Always wrap your main logic in if __name__ == "__main__": — this is mandatory on Windows to prevent child processes from re-executing your entire script.
• If your complex_function already spawns multiple processes internally, reduce the pool size (e.g., processes=multiprocessing.cpu_count()//2) to avoid overwhelming your system.
• imap_unordered is better if you don't need results in the original variable order, as it returns data as soon as each process finishes.

Approach 2: Manual Process Creation (For Fine-Grained Control) #

If you need more control over individual processes (like monitoring status or assigning specific resources), you can create Process instances manually. Use a Queue to collect results since processes don't share memory directly.

import multiprocessing

# Your existing complex function
def complex_function(variable, result_queue):
    # Original multiprocessing logic here...
    result = variable * 2
    # Send results back via queue
    result_queue.put((variable, result))

if __name__ == "__main__":
    variables = range(10, 101)
    result_queue = multiprocessing.Queue()
    processes = []
    
    # Create a process for each variable
    for var in variables:
        process = multiprocessing.Process(
            target=complex_function,
            args=(var, result_queue)
        )
        processes.append(process)
        process.start()
    
    # Wait for all processes to finish
    for process in processes:
        process.join()
    
    # Collect and sort results (queue order isn't guaranteed)
    results = []
    while not result_queue.empty():
        results.append(result_queue.get())
    results.sort(key=lambda x: x[0])
    
    # Output results
    for var, res in results:
        print(f"Variable {var} result: {res}")

Key Notes for This Approach: #

• Avoid creating too many processes at once (90 variables would mean 90 processes) — this can slow down your system due to context switching. Stick to Pool if you're processing a large range.
• Use Queue or other IPC mechanisms to share data between processes; regular variables won't work across process boundaries.

Critical Final Reminder: #

If your complex_function already uses multiprocessing heavily, don't overdo the outer process count. Nested multiprocessing can lead to resource exhaustion or unexpected crashes. Test with a small range (e.g., 10 to 12) first to validate everything works before scaling to 10-100.

内容的提问来源于stack exchange，提问作者ZHANG Juenjie