Python Multiprocessing Pool类运行卡住，求问题排查及解决方法

Problem Description #

I was trying to use multiprocessing in another task but it wouldn't run properly. To verify if I was using the Pool class correctly, I wrote this simplified test code. However, the program still gets stuck indefinitely, and I have to restart the IPython shell to continue working. What did I do wrong?

from multiprocessing import Pool
def square(x):
    sq = x**2
    return sq
def main():
    x1 = [1,2,3,4]
    pool = Pool()
    result = pool.map( square, x1 )
    print(result)
if __name__ == '__main__':
    main()

Answer #

Hey, I’ve run into this exact problem before! This is a common headache when using multiprocessing.Pool in IPython (or Jupyter Notebook) interactive environments. Let me break down why it’s happening and how to fix it:

Why It’s Getting Stuck

On Unix-like systems, the default start method for multiprocessing is fork. When you create a Pool in IPython, the child processes spawned via fork try to re-import your interactive session’s namespace to set up their environment. Unlike regular .py scripts (where if __name__ == '__main__': stops the main code from running again in child processes), IPython’s interactive setup doesn’t handle this cleanly. This leads to a deadlock where child processes can’t initialize properly, causing your program to hang.

Fixes to Try

1. Run the Code as a Standalone Script

The simplest fix is to save your code into a separate .py file (like test_pool.py) and run it from your terminal with:

python test_pool.py

This way, the if __name__ == '__main__': guard works as intended, and child processes won’t re-execute your main logic. You’ll get the expected output [1, 4, 9, 16] without any hangs.

2. Change the Process Start Method in IPython

If you really need to run this directly in IPython, switch to the spawn start method. This creates fresh Python processes instead of forking the current one, avoiding the namespace re-import issue. Here’s the modified code:

from multiprocessing import Pool, set_start_method

def square(x):
    sq = x**2
    return sq

def main():
    x1 = [1, 2, 3, 4]
    # Set spawn as the start method to avoid fork-related deadlocks
    try:
        set_start_method('spawn')
    except RuntimeError:
        # Ignore if the start method was already set earlier
        pass
    
    pool = Pool()
    result = pool.map(square, x1)
    print(result)
    
    # Clean up the pool to release resources properly
    pool.close()
    pool.join()

if __name__ == '__main__':
    main()

You can either paste this modified code into IPython, or save it to a file and run it using IPython’s %run magic command:

%run test_pool.py

Adding pool.close() and pool.join() is also a good practice—it ensures all child processes finish their tasks and free up resources, which helps prevent hangs in interactive environments.

内容的提问来源于stack exchange，提问作者Nick-H