使用Pybind11的Python/C++多线程代码为何出现死锁？

问题背景 #

将Python项目的部分计算卸载到C以提升性能，使用Pybind11作为桥接工具。在多C线程调用Python回调函数修改Python对象数值时，出现死锁问题，简化代码如下：

Python代码 #

import example
import threading

class MyClass:
    def __init__(self, value=0):
        self.value = value
        self.lock = threading.Lock()

    def python_callback(self, arg1: float, arg2: float) -> float:
        with self.lock:
            self.value += arg1
            self.value += arg2
        return float(self.value)

myclass = MyClass()

def cpp_thread():
    example.call_cpp_thread(myclass, 1.0, 1.0, 10000)

def python_thread():
    for i in range(40000):
        myclass.python_callback(1, 1)

thread1 = threading.Thread(target=cpp_thread)
thread2 = threading.Thread(target=python_thread)
thread1.start()
thread2.start()

thread1.join()
thread2.join()

print(myclass.value)

C++代码 #

#include <pybind11/pybind11.h>
#include <pybind11/functional.h>
#include <pybind11/stl.h>
#include <thread>

namespace py = pybind11;

void call_python_callback(py::object obj, float arg1, float arg2, int calltime) {
    for (int i = 0; i < calltime; i++) {
        obj.attr("python_callback")(arg1, arg2);
    }
}

void call_cpp_thread(py::object obj, float arg1, float arg2, int calltime) {
    std::thread t1(call_python_callback, obj, arg1, arg2, calltime);
    std::thread t2(call_python_callback, obj, arg1, arg2, calltime);
    std::thread t3(call_python_callback, obj, arg1, arg2, calltime);
    std::thread t4(call_python_callback, obj, arg1, arg2, calltime);
    t1.join();
    t2.join();
    t3.join();
    t4.join();
}

PYBIND11_MODULE(example, m) {
    m.def("call_python_callback", &call_python_callback, "Call a Python callback",
          py::arg("obj"), py::arg("arg1"), py::arg("arg2"), py::arg("calltime"));
    m.def("call_cpp_thread", &call_cpp_thread, "Call cpp thread",
          py::arg("obj"), py::arg("arg1"), py::arg("arg2"), py::arg("calltime"));
}

死锁原因分析 #

死锁源于GIL（全局解释器锁）与Python线程锁（threading.Lock）的交叉持有：

1. Python线程（thread2）进入python_callback后，先获取self.lock，Python解释器会定期自动释放GIL以调度其他线程，但此时self.lock仍被该线程持有。
2. 某个C线程获取到GIL后，尝试调用python_callback，进入函数后尝试获取self.lock，但锁已被Python线程持有，于是C线程阻塞在锁上，同时持续持有GIL。
3. Python线程需要重新获取GIL才能继续执行并释放self.lock，但GIL被阻塞的C++线程持有，形成循环等待，最终触发死锁。

修复方案 #

方案一：将同步逻辑移至C++侧（推荐） #

使用C++的std::mutex保护对Python对象的访问，避免GIL与Python锁的交叉冲突：

修改后的C++代码：

#include <pybind11/pybind11.h>
#include <pybind11/functional.h>
#include <pybind11/stl.h>
#include <thread>
#include <mutex>

namespace py = pybind11;

// 全局互斥锁，保护对Python对象的并发访问
std::mutex callback_mutex;

void call_python_callback(py::object obj, float arg1, float arg2, int calltime) {
    for (int i = 0; i < calltime; i++) {
        std::lock_guard<std::mutex> lock(callback_mutex);
        py::gil_scoped_acquire acquire; // 手动获取GIL
        obj.attr("python_callback")(arg1, arg2);
        // GIL会在acquire对象销毁时自动释放
    }
}

void call_cpp_thread(py::object obj, float arg1, float arg2, int calltime) {
    std::thread t1(call_python_callback, obj, arg1, arg2, calltime);
    std::thread t2(call_python_callback, obj, arg1, arg2, calltime);
    std::thread t3(call_python_callback, obj, arg1, arg2, calltime);
    std::thread t4(call_python_callback, obj, arg1, arg2, calltime);
    t1.join();
    t2.join();
    t3.join();
    t4.join();
}

PYBIND11_MODULE(example, m) {
    m.def("call_python_callback", &call_python_callback, "Call a Python callback",
          py::arg("obj"), py::arg("arg1"), py::arg("arg2"), py::arg("calltime"));
    m.def("call_cpp_thread", &call_cpp_thread, "Call cpp thread",
          py::arg("obj"), py::arg("arg1"), py::arg("arg2"), py::arg("calltime"));
}

修改后的Python代码（移除threading.Lock）：

import example
import threading

class MyClass:
    def __init__(self, value=0):
        self.value = value

    def python_callback(self, arg1: float, arg2: float) -> float:
        self.value += arg1
        self.value += arg2
        return float(self.value)

myclass = MyClass()

def cpp_thread():
    example.call_cpp_thread(myclass, 1.0, 1.0, 10000)

def python_thread():
    for i in range(40000):
        myclass.python_callback(1, 1)

thread1 = threading.Thread(target=cpp_thread)
thread2 = threading.Thread(target=python_thread)
thread1.start()
thread2.start()

thread1.join()
thread2.join()

print(myclass.value)

方案二：确保Python锁的持有周期不脱离GIL控制 #

如果必须保留Python侧的锁，可以在C++调用回调时，在整个循环周期内持有GIL，避免Python线程抢占GIL后持有锁：

修改后的C++代码：

void call_python_callback(py::object obj, float arg1, float arg2, int calltime) {
    py::gil_scoped_acquire acquire; // 整个循环期间持有GIL
    for (int i = 0; i < calltime; i++) {
        obj.attr("python_callback")(arg1, arg2);
    }
}

注意：此方案会让C++线程长时间持有GIL，可能降低Python线程的执行效率，仅适合回调执行时间较短的场景。

验证结果 #

修复后运行代码，死锁问题消失，最终输出的myclass.value应为预期的(4*10000 + 40000)*2 = 160000。

内容的提问来源于stack exchange，提问作者Chu Pika