使用pybind11在递归Python代码中调用hpx异步函数遇报错求助

我想在Python递归函数里调用通过pybind11绑定的HPX异步函数，但运行失败且无法定位原因。以下是我的HPX异步函数绑定代码、Python递归调用代码、报错信息，尝试使用PyGILState_Ensure获取GIL后问题仍未解决。

原始C++绑定代码 #

hpx::shared_future<int> myAsyncFunction(py::function f, py::args args)
{
    return hpx::async([f, args]() {
        py::gil_scoped_acquire acquire;
        py::object result = f(*args);
        return result.cast<int>();
    });
}

Python递归代码 #

import test
import pdb

test.init_hpx()

def fibonacci(n):
    # pdb.set_trace()
    if(n<2):
        return n
    
    n1 = test.new_async(fibonacci,n - 1)
    n2 = test.new_async(fibonacci,n - 2)

    if n1.is_ready() and n2.is_ready():
        return n1.get() + n2.get()

print(fibonacci(10))

报错信息 #

None
Fatal Python error: PyThreadState_Get: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)
Python runtime state: initialized

Thread 0x00007f508b7fe640 (most recent call first):

Thread 0x00007f509353a000 (most recent call first):
File "/home/matrix/pyhpx/python_hpx/build/fib.py", line 17 in

尝试修改后的C++代码 #

hpx::shared_future<int> myAsyncFunction(py::function f, py::args args) {
    return hpx::async([f, args]() {
        PyGILState_STATE gstate = PyGILState_Ensure(); // Acquire the GIL
        py::object result = f(*args);
        // Release the GIL
        PyGILState_Release(gstate);

        return result.cast<int>();
    });
}

问题分析与解决思路 #

1. GIL管理逻辑错误
   手动调用PyGILState_Ensure/PyGILState_Release时，result.cast<int>()操作被放在了GIL释放之后，而该操作需要访问Python对象，必须在GIL持有状态下执行。同时，pybind11的py::gil_scoped_acquire已经封装了完整的GIL管理逻辑，混用手动API反而容易出错。

   修正后的C++代码：

   hpx::shared_future<int> myAsyncFunction(py::function f, py::args args)
   {
       return hpx::async([f, args]() {
           py::gil_scoped_acquire acquire;
           py::object result = f(*args);
           int res = result.cast<int>(); // 转换操作放在GIL持有块内
           return res;
       });
   }
   

2. Python端异步等待逻辑缺陷
   当前Python代码中，仅当两个异步任务同时就绪时才返回结果，否则函数会隐式返回None（这也是报错前输出None的原因）。需要改为阻塞等待任务完成，或者循环等待直到就绪：

   修正后的Python代码：

   import test
   
   test.init_hpx()
   
   def fibonacci(n):
       if n < 2:
           return n
       
       n1 = test.new_async(fibonacci, n - 1)
       n2 = test.new_async(fibonacci, n - 2)
       
       # 阻塞等待两个任务完成后再求和
       return n1.get() + n2.get()
   
   print(fibonacci(10))
   

内容的提问来源于stack exchange，提问作者Shubham kumar