Litellm v1.49.7搭配Celery --pool=gevent调用LLM报错求助

我想用litellm==1.49.7结合Celery的--pool=gevent实现LLM模型的并行调用，但遇到了运行时错误。试过pool=threads和pool=prefork都能正常运行，但业务有高IO操作，gevent更适合我的场景。

代码片段 #

response = loop.create_task(completion(
                                    model=SELECTED_AI_MODEL,
                                    messages=messages,
                                    temperature=0,
                                    max_tokens=max_tokens,
                                    top_p=0.1,
                                    stream=True,
                                    max_retries=2,
                                    timeout=60,
                                    response_format={"type": "json_object"},
                                ))

错误信息 #

Exception in thread 
[2024-10-18 20:54:12,023: WARNING/MainProcess] Thread-7 (run_success_logging_and_cache_storage)
[2024-10-18 20:54:12,023: WARNING/MainProcess] :
[2024-10-18 20:54:12,023: WARNING/MainProcess] Traceback (most recent call last):
[2024-10-18 20:54:12,023: WARNING/MainProcess]   File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
[2024-10-18 20:54:12,025: WARNING/MainProcess]     
[2024-10-18 20:54:12,026: WARNING/MainProcess] self.run()
[2024-10-18 20:54:12,026: WARNING/MainProcess]   File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1010, in run
[2024-10-18 20:54:12,026: WARNING/MainProcess]     
[2024-10-18 20:54:12,027: WARNING/MainProcess] self._target(*self._args, **self._kwargs)
[2024-10-18 20:54:12,027: WARNING/MainProcess]   File "/Users/users/OFFICE_PROJECTS/ai/services/venv/lib/python3.12/site-packages/litellm/utils.py", line 7854, in run_success_logging_and_cache_storage
[2024-10-18 20:54:12,031: WARNING/MainProcess]     
[2024-10-18 20:54:12,031: WARNING/MainProcess] asyncio.run(
[2024-10-18 20:54:12,031: WARNING/MainProcess]   File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/runners.py", line 190, in run
[2024-10-18 20:54:12,032: WARNING/MainProcess]     
[2024-10-18 20:54:12,032: WARNING/MainProcess] raise RuntimeError(
[2024-10-18 20:54:12,033: WARNING/MainProcess] RuntimeError
[2024-10-18 20:54:12,033: WARNING/MainProcess] : 
[2024-10-18 20:54:12,033: WARNING/MainProcess] asyncio.run() cannot be called from a running event loop
[2024-10-18 20:54:12,034: WARNING/MainProcess] /usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py:1075: RuntimeWarning: coroutine 'Logging.async_success_handler' was never awaited
  self._invoke_excepthook(self)

解决方案 #

1. 禁用LiteLLM的自动日志/缓存异步处理 #

错误根源是LiteLLM内部的run_success_logging_and_cache_storage函数在已有事件循环的环境下调用了asyncio.run()，和gevent协程模型冲突。通过配置关闭相关功能：

import litellm

# 禁用成功日志回调
litellm.success_callback = []
# 或者禁用缓存（如果不需要）
litellm.cache = None

2. 确保Celery gevent环境的补丁生效 #

启动Celery时添加环境变量并提前打gevent补丁：

启动命令

GEVENT_SUPPORT=True celery -A your_app worker --pool=gevent --concurrency=10

代码中提前打补丁

from gevent import monkey
monkey.patch_all()

3. 调整LiteLLM调用方式 #

• 关闭流式输出（如果允许）：改用同步调用接口，避免asyncio事件循环冲突：

response = completion(
    model=SELECTED_AI_MODEL,
    messages=messages,
    temperature=0,
    max_tokens=max_tokens,
    top_p=0.1,
    stream=False,
    max_retries=2,
    timeout=60,
    response_format={"type": "json_object"},
)

• 保留流式输出：用gevent的异步工具包装调用：

from gevent import async

response = async(completion(
    model=SELECTED_AI_MODEL,
    messages=messages,
    temperature=0,
    max_tokens=max_tokens,
    top_p=0.1,
    stream=True,
    max_retries=2,
    timeout=60,
    response_format={"type": "json_object"},
))

4. 升级LiteLLM版本 #

该问题可能在后续版本中被修复，尝试升级到最新稳定版：

pip install --upgrade litellm

内容的提问来源于stack exchange，提问作者nawij itrahg