Python中Popen创建进程后残留额外进程问题咨询（含multiprocessing manager）

背景与问题 #

我用以下简化后的LocalEngine类管理进程：

class LocalEngine:

    def __init__(self):
        name_to_pid = {}

    def run_instance(self, name):
        args = ... # 根据参数计算Popen的args
        with open(f"out-{name}", 'a') as out, open(f"err-{name}", 'a') as err:
            pid = \
                subprocess.Popen(args, stdout=out, stderr=err, shell=False).pid
            print(f"Created process {pid}")
            self.name_to_pid[name] = pid

    def kill_instance(self, name):
        pid = self.name_to_pid[name]
        print(f"Killing process {pid}")
        psutil.Process(pid).kill()

系统中只有一个LocalEngine实例engine，所有进程都通过engine.run_instance创建，最终用engine.kill_instance终止。

Ubuntu下主脚本输出如下：

Created process 9676
Created process 9703
Killing process 9703
Killing process 9676

但进程创建完成后，执行ps auxfww发现每个Popen调用都生成了两个进程（已指定shell=False），相关输出：

root     27576  0.0  0.0    896    80 ?        S    10:33   0:00  \_ /init
meir     27577  0.0  0.0  10688  5944 pts/2    Ss   10:33   0:00      \_ -bash
meir      9667  2.2  0.0  24296 16956 pts/2    S+   15:26   0:00          \_ python -m examples.agent_assignment.run_server
meir      9668  0.3  0.0 392712 16732 pts/2    Sl+  15:26   0:00              \_ python -m examples.agent_assignment.run_server
meir      9676  1.9  0.0  24016 16340 pts/2    S+   15:26   0:00              \_ python -m src.run_backup 127.0.1.1 57785 server-1
meir      9677  3.7  0.0 392784 14648 pts/2    Sl+  15:26   0:00              |   \_ python -m src.run_backup 127.0.1.1 57785 server-1
meir      9703  8.0  0.0  28364 21084 pts/2    S+   15:26   0:00              \_ python -m examples.agent_assignment.run_client 127.0.1.1 57785 client-1 2
meir      9704 10.0  0.0 765568 18912 pts/2    Sl+  15:26   0:00                  \_ python -m examples.agent_assignment.run_client 127.0.1.1 57785 client-1 2

其中run_server是主脚本，两次Popen分别启动run_backup和run_client脚本。

进程终止后，ps输出仍有残留：

root     27575  0.0  0.0    896    80 ?        Ss   10:33   0:00 /init
root     27576  0.0  0.0    896    80 ?        S    10:33   0:00  \_ /init
meir     27577  0.0  0.0  10688  5944 pts/2    Ss+  10:33   0:00      \_ -bash
meir      9677  0.1  0.0 466516 14736 pts/2    Sl   15:26   0:00      \_ python -m src.run_backup 127.0.1.1 57785 server-1
meir      9704  0.0  0.0 749176 21088 pts/2    Sl   15:26   0:00      \_ python -m examples.agent_assignment.run_client 127.0.1.1 57785 client-1 2

尝试修改后的代码 #

根据建议修改了kill_instance方法（因跨进程反序列化需求，只能存储PID，无法保存进程对象）：

def kill_instance(self, name):
        # 查看进程树命令：ps auxfww
        pid = self.name_to_pid[name]
        print(f"Terminating process {pid}")
        proc = psutil.Process(pid)
        proc.terminate()
        print(f"Waiting for process {proc} to terminate")
        try:
            proc.wait(timeout=10)
        except subprocess.TimeoutExpired:
            print(f"Process {pid} did not terminate in time, killing it")
            proc.kill()

修改后未触发超时异常，但仍有进程残留。

问题更新 #

已确认额外进程是由multiprocessing.Manager启动导致的，现在需要解决如何同时终止主进程和manager进程的问题。

内容的提问来源于stack exchange，提问作者AlwaysLearning