Windows下C++调用Python Multiprocessing致进程无限创建问题求助

在Windows系统开发项目，架构为C主程序通过Python C API调用含Multiprocessing的Python函数。单进程Python代码调用正常，但只要启用Multiprocessing（哪怕只创建1个进程），程序就会持续生成新的C主进程；单独运行Python脚本时，Multiprocessing可正常工作。

现象总结 #

• C++主程序调用单进程Python代码时运行正常；
• C++主程序调用含多进程的Python代码时，持续创建新进程，Python代码未执行；
• 单独运行多进程Python脚本时，可正常并行执行。

排查过程 #

在Python代码中打印sys.executable和os.getpid()后发现，调用p1.start()时会递归创建新的C++主进程，推测问题与Multiprocessing识别主/子进程的机制有关（类似独立Python环境中if __name__ == '__main__'的必要性）。

解决方法 #

通过修改Python代码解决问题：设置sys.argv指定当前脚本路径，调用mp.set_executable指定Python解释器路径，C++代码无需修改。

初始start.py代码 #

import time
import multiprocessing as mp


def do_something():
    print('Sleeping 1 second...')
    time.sleep(1)
    print('Done Sleeping...')
    
def benchmark():
    start = time.perf_counter()

    p1 = mp.Process(target=do_something)
    p2 = mp.Process(target=do_something)
    p1.start()
    #p2.start()
    p1.join()
    #p2.join()

    finish = time.perf_counter()

    print(f'Finished in {round(finish - start,2)} second(s)')
 
if __name__ == '__main__':
    benchmark()

callPythonFromCpp.cpp代码 #

#include <Windows.h>
#include <iostream>
#include <string>
#include <Python.h>


using namespace std;

void CallPython(string PythonModuleName, string PythonFunctionName)
{
    char* funcname = new char[PythonFunctionName.length() + 1];
    strcpy_s(funcname, PythonFunctionName.length() + 1, PythonFunctionName.c_str());

    char* modname = new char[PythonModuleName.length() + 1];
    strcpy_s(modname, PythonModuleName.length() + 1, PythonModuleName.c_str());

    
    // Initialize the Python interpreter 
    Py_Initialize();

    PyObject* my_module = PyImport_ImportModule(modname);

    PyObject* my_function = PyObject_GetAttrString(my_module, funcname);

    // Call a callable Python object callable, with arguments given by the tuple args. 
    // If no arguments are needed, then args can be NULL.

    PyObject* my_result = PyObject_CallObject(my_function, NULL);

    // Undo all initializations made by Py_Initialize() and subsequent use of Python/C API functions, 
    // and destroy all sub-interpreters (see Py_NewInterpreter() below) that were created and not yet 
    // destroyed since the last call to Py_Initialize(). Ideally, this frees all memory allocated by the Python interpreter.
    Py_Finalize();

    delete[] funcname;
    delete[] modname;
}
int main()
{
    CallPython("start", "benchmark");
    system("pause");
    return 0;
}

修复后的start.py代码 #

import sys
import time
import multiprocessing as mp


def do_something():
    print('Sleeping 1 second...')
    time.sleep(1)
    print('Done Sleeping...')
    
def benchmark():
    sys.argv = [r'C:\path_to_this\start.py']
    mp.set_executable(r'C:\path_to_Python_install\python.exe')

    start = time.perf_counter()

    p1 = mp.Process(target=do_something)
    p2 = mp.Process(target=do_something)
    p1.start()
    p2.start()
    p1.join()
    p2.join()

    finish = time.perf_counter()

    print(f'Finished in {round(finish - start,2)} second(s)')
 
if __name__ == '__main__':
    benchmark()

注：修复后的代码需补充导入sys模块，否则会因未定义sys变量报错。

内容的提问来源于stack exchange，提问作者panzerkiller