Python并行函数重复运行内存占用攀升问题及解决方案咨询

First off, let’s answer your two questions directly:

1. Is the memory growth caused by parallelization? #

Yes, it’s directly tied to how you’re managing your parallelization setup. The issue isn’t parallelization itself—it’s that you’re creating a new multiprocessing.Pool (with 4 child processes) in every loop iteration, but never properly shutting those processes down. The del results line only cleans up the result list in your main process, but the child processes from each Pool stick around in the background, hoarding memory and causing that gradual growth you’re seeing.

2. How to stop the memory growth? #

You have two straightforward fixes—pick whichever fits your use case best:

Option 1: Reuse the same Pool across all iterations #

Instead of spawning new processes every loop, initialize the Pool once before the loop runs. This way, you’ll reuse the same set of child processes for all 10 runs, eliminating the repeated overhead of creating and discarding processes. Using a with statement ensures the Pool is automatically cleaned up when you’re done:

import multiprocessing as mp
import numpy as np
from tqdm import tqdm

def function(x):
    return x*2

test_array = np.arange(0, 1e4, 1)

# Create Pool once outside the loop
with mp.Pool(processes=4) as pool:
    for i in range(10):
        results = list(tqdm(pool.imap(function, test_array), total=len(test_array)))
        # Filter NaNs more efficiently (see bonus tip below)
        results = np.array(results)
        results = results[~np.isnan(results)].tolist()

Option 2: Properly clean up the Pool inside each loop #

If you need a fresh Pool every iteration (e.g., your function has state that needs resetting), make sure to explicitly shut down the Pool after each use. Call pool.close() (to stop accepting new tasks) and pool.join() (to wait for all child processes to exit) before moving to the next iteration:

import multiprocessing as mp
import numpy as np
from tqdm import tqdm

def function(x):
    return x*2

test_array = np.arange(0, 1e4, 1)
for i in range(10):
    pool = mp.Pool(processes=4)
    results = list(tqdm(pool.imap(function, test_array), total=len(test_array)))
    # Filter NaNs efficiently
    results = np.array(results)
    results = results[~np.isnan(results)].tolist()
    
    # Clean up the Pool fully
    pool.close()
    pool.join()
    
    del results

Bonus Tip: Optimize NaN filtering #

Your original line results = [x for x in results if str(x) != 'nan'] is inefficient—converting numbers to strings to check for NaNs creates unnecessary overhead. Using numpy’s vectorized operations is faster and uses less memory, as shown in the code examples above.

内容的提问来源于stack exchange，提问作者user8188120