multiprocessing.Manager致Popen.communicate挂起，求信号终止解决方法

使用multiprocessing.Manager后，调用subprocess.Popen.terminate()或subprocess.Popen.kill()无法干净终止Python子进程——原因是Manager会在后台创建一个用于通信的子进程，父进程终止时该后台进程不会自动清理，最终导致主进程挂起。

问题演示代码 #

"""Multiprocess manager hang test."""
import multiprocessing
import subprocess
import sys
import time


def launch_and_read_process():
    proc = subprocess.Popen(
        [
            "python",
            sys.argv[0],
            "run_unkillable"
        ],
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
    )

    # 等待进程启动并输出内容
    time.sleep(3)

    status = proc.poll()
    print("poll() is", status)

    print("Terminating")
    assert proc.returncode is None
    proc.terminate()
    exit_code = proc.wait()
    print("Got exit code", exit_code)
    stdout, stderr = proc.communicate()
    print("Got output", stdout.decode("utf-8"))


def run_unkillable():
    # 注释掉Manager创建代码即可正常终止
    manager = multiprocessing.Manager()
    d = manager.dict()
    d["foo"] = "bar"
    print("This is an example output", flush=True)
    time.sleep(999)


def main():
    mode = sys.argv[1]
    print("Doing subrouting", mode)
    func = globals().get(mode)
    func()


if __name__ == "__main__":
    main()

执行命令：

python test-script.py launch_and_read_process

未使用multiprocessing.Manager时的正常输出 #

Doing subrouting launch_and_read_process
poll() is None
Terminating
Got exit code -15
Got output Doing subrouting run_unkillable
This is an example output

使用multiprocessing.Manager导致挂起时的输出 #

Doing subrouting launch_and_read_process
poll() is None
Terminating
Got exit code -15

解决方案 #

1. 显式关闭Manager并清理后台进程

这是最规范的做法，适用于可修改子进程代码的场景。通过上下文管理器自动管理生命周期，或手动调用shutdown()确保后台通信进程被终止。

修改run_unkillable函数：

def run_unkillable():
    # 用上下文管理器自动关闭Manager
    with multiprocessing.Manager() as manager:
        d = manager.dict()
        d["foo"] = "bar"
        print("This is an example output", flush=True)
        time.sleep(999)

或手动关闭：

def run_unkillable():
    manager = multiprocessing.Manager()
    try:
        d = manager.dict()
        d["foo"] = "bar"
        print("This is an example output", flush=True)
        time.sleep(999)
    finally:
        # 显式终止后台通信进程
        manager.shutdown()

2. 向整个进程组发送终止信号

若无法修改子进程代码，可在父进程中向子进程的整个进程组发送信号，确保Manager的后台进程也被终止。需让子进程成为新会话领头进程，再通过进程组ID统一发送信号。

修改launch_and_read_process函数：

import os
import signal

def launch_and_read_process():
    proc = subprocess.Popen(
        [
            "python",
            sys.argv[0],
            "run_unkillable"
        ],
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        start_new_session=True  # 让子进程成为新会话领头进程，统一管理进程组
    )

    time.sleep(3)

    status = proc.poll()
    print("poll() is", status)

    print("Terminating")
    assert proc.returncode is None
    try:
        # 向整个进程组发送SIGTERM信号
        os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
    except ProcessLookupError:
        # 进程已提前退出，忽略错误
        pass
    exit_code = proc.wait()
    print("Got exit code", exit_code)
    stdout, stderr = proc.communicate()
    print("Got output", stdout.decode("utf-8"))

3. 替换multiprocessing.Manager为其他IPC方式

若业务场景允许，避免使用multiprocessing.Manager，改用更轻量的进程间通信机制：

• 使用multiprocessing.Queue或multiprocessing.Pipe实现进程通信
• 若仅需共享简单数据，使用multiprocessing.Array或multiprocessing.Value
• 自定义基于共享内存的数据结构，减少额外进程开销

内容的提问来源于stack exchange，提问作者Mikko Ohtamaa