Python3.11+Celery+Gevent运行报错：returned NULL without setting an exception

问题背景 #

在容器环境中使用 Python 3.11 搭配 Celery、Gevent，以 Redis 作为 Broker 运行 Worker 时，触发以下 SystemError 异常，目前降级到 Python 3.10 可临时解决问题，现寻求无需降级 Python 的替代方案。

异常日志 #

[2023-08-04 11:10:28,805] [WARNING] [Dummy-748] [log.py:232 - write()] Traceback (most recent call last):
[2023-08-04 11:10:28,806] [WARNING] [Dummy-748] [log.py:232 - write()]   File "src/gevent/_waiter.py", line 122, in gevent._gevent_c_waiter.Waiter.switch
[2023-08-04 11:10:28,806] [WARNING] [Dummy-748] [log.py:232 - write()] SystemError: <built-in method switch of gevent._gevent_cgreenlet.Greenlet object at 0x7fcdb37de700> returned NULL without setting an exception
[2023-08-04 11:10:28,806] [WARNING] [Dummy-748] [log.py:232 - write()] 2023-08-04T11:10:28Z
[2023-08-04 11:10:28,806] [WARNING] [Dummy-748] [log.py:232 - write()]
[2023-08-04 11:10:28,806] [WARNING] [Dummy-748] [log.py:232 - write()] <built-in method switch of gevent._gevent_cgreenlet.Greenlet object at 0x7fcdb37de700> failed with SystemError
[2023-08-04 11:10:28,806] [ERROR] [MainThread] [consumer.py:248 - perform_pending_operations()] Pending callback raised: SystemError('<built-in method switch of gevent._gevent_cgreenlet.Greenlet object at 0x7fcdb37de700> returned NULL without setting an exception')
Traceback (most recent call last):
  File "/home/.local/lib/python3.11/site-packages/celery/worker/consumer/consumer.py", line 246, in perform_pending_operations
    self._pending_operations.pop()()
  File "/home/.local/lib/python3.11/site-packages/vine/promises.py", line 160, in __call__
    return self.throw()
           ^^^^^^^^^^^^
  File "/home/.local/lib/python3.11/site-packages/vine/promises.py", line 157, in __call__
    retval = fun(*final_args, **final_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.local/lib/python3.11/site-packages/kombu/message.py", line 131, in ack_log_error
    self.ack(multiple=multiple)
  File "/home/.local/lib/python3.11/site-packages/kombu/message.py", line 126, in ack
    self.channel.basic_ack(self.delivery_tag, multiple=multiple)
  File "/home/.local/lib/python3.11/site-packages/kombu/transport/virtual/base.py", line 670, in basic_ack
    self.qos.ack(delivery_tag)
  File "/home/.local/lib/python3.11/site-packages/kombu/transport/redis.py", line 379, in ack
    self._remove_from_indices(delivery_tag).execute()
  File "/home/.local/lib/python3.11/site-packages/redis/client.py", line 2120, in execute
    return conn.retry.call_with_retry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.local/lib/python3.11/site-packages/redis/retry.py", line 46, in call_with_retry
    return do()
           ^^^^
  File "/home/.local/lib/python3.11/site-packages/redis/client.py", line 2121, in <lambda>
    lambda: execute(conn, stack, raise_on_error),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.local/lib/python3.11/site-packages/redis/client.py", line 1975, in _execute_transaction
    self.parse_response(connection, "_")
  File "/home/.local/lib/python3.11/site-packages/redis/client.py", line 2060, in parse_response
    result = Redis.parse_response(self, connection, command_name, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.local/lib/python3.11/site-packages/redis/client.py", line 1286, in parse_response
    response = connection.read_response()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.local/lib/python3.11/site-packages/redis/connection.py", line 882, in read_response
    response = self._parser.read_response(disable_decoding=disable_decoding)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.local/lib/python3.11/site-packages/redis/connection.py", line 349, in read_response
    result = self._read_response(disable_decoding=disable_decoding)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.local/lib/python3.11/site-packages/redis/connection.py", line 359, in _read_response
    raw = self._buffer.readline()
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.local/lib/python3.11/site-packages/redis/connection.py", line 262, in readline
    self._read_from_socket()
  File "/home/.local/lib/python3.11/site-packages/redis/connection.py", line 212, in _read_from_socket
    data = self._sock.recv(socket_read_size)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.local/lib/python3.11/site-packages/gevent/_socketcommon.py", line 666, in recv
    self._wait(self._read_event)
  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 304, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
  File "src/gevent/_waiter.py", line 122, in gevent._gevent_c_waiter.Waiter.switch
SystemError: <built-in method switch of gevent._gevent_cgreenlet.Greenlet object at 0x7fcdb37de700> returned NULL without setting an exception

可行替代解决方案 #

1. 升级 Gevent 到兼容 Python 3.11 的版本 #

该异常源于 Gevent 的 C 扩展在 Python 3.11 协程切换逻辑中的兼容性 bug，Gevent 22.10.0 及以上版本已正式支持 Python 3.11，修复了底层切换问题。执行升级命令：

pip install --upgrade gevent>=22.10.0

2. 替换 Redis 客户端的连接实现 #

问题出在 Redis 客户端 socket 读取与 Gevent monkey-patch 的交互冲突，可自定义 Gevent 兼容的 Redis 连接类：

from redis.connection import Connection
from gevent.socket import socket

class GeventRedisConnection(Connection):
    def _connect(self):
        self._sock = socket()
        self._sock.connect((self.host, self.port))

然后在 Celery 配置中指定该连接类：

CELERY_REDIS_CONNECTION_CLASS = 'your.module.path.GeventRedisConnection'

3. 调整 Gevent 的 Monkey-Patching 范围 #

若业务允许，可跳过 socket 模块的 monkey-patch，避免协程切换冲突：

from gevent import monkey
monkey.patch_all(socket=False)

注意：此方式可能降低 Gevent 异步性能，需结合业务场景测试验证。

内容的提问来源于stack exchange，提问作者DanielM