Django中使用grequests批量调用API时应用意外退出求助

问题背景 #

在Django项目中使用grequests实现100个API的批量异步调用，代码在Jupyter Notebook中测试完全正常，但通过python manage.py runserver启动开发服务器后，执行完一批（10个）API调用后应用会无征兆退出。已知SSL证书验证警告（InsecureRequestWarning）并非导致退出的原因。

问题代码 #

class BatchCall:
    def __init__(self, urls, BO_urls):
        self.urls = urls
        self.BO_urls = BO_urls

    def exception(self, request, exception):
        print("Problem: {}: {}".format(request.url, exception))

    def asyncCall_release(self):
        return grequests.map(
            (grequests.get(u, auth=HTTPBasicAuth(username, password), verify=False) for u in self.urls),
            exception_handler=self.exception, size=10)

    def asyncCall_BO(self):
        return grequests.map(
            (grequests.get(u, auth=HTTPBasicAuth(username, password), verify=False) for u in self.BO_urls),
            exception_handler=self.exception, size=10)

test = BatchCall(urls, BO_urls)
results_release = test.asyncCall_release()
results_BO = test.asyncCall_BO()

原因分析 #

grequests基于gevent实现异步，而Django开发服务器（runserver）的默认线程模型、自动重载机制与gevent的协程调度存在冲突：

• gevent通过猴子补丁修改标准库实现协程，但Django开发服务器启动时未被patch，导致协程和线程的事件循环冲突，触发进程退出
• runserver的自动重载功能会重启进程，干扰gevent的协程上下文

解决方案 #

1. 提前添加gevent猴子补丁 #

在项目的manage.py最顶部添加gevent的猴子补丁，确保所有标准库被patch，需放在所有其他import之前：

from gevent import monkey
monkey.patch_all()

# 以下是原有manage.py的代码
import os
import sys

def main():
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'your_project.settings')
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(...) from exc
    execute_from_command_line(sys.argv)

if __name__ == '__main__':
    main()

2. 禁用开发服务器自动重载 #

启动服务器时添加--noreload参数，避免自动重载机制破坏协程上下文：

python manage.py runserver --noreload

3. 调整并发数 #

适当降低grequests.map的size参数（比如改为5），减少短时间内的协程数量，避免资源竞争：

return grequests.map(
    (grequests.get(u, auth=HTTPBasicAuth(username, password), verify=False) for u in self.urls),
    exception_handler=self.exception, size=5)

4. 改用Django原生异步方案（推荐） #

如果上述方法仍存在兼容性问题，推荐使用Django 3.1+原生异步支持结合aiohttp实现，更贴合Django生态：

import asyncio
import aiohttp
from django.http import HttpResponse
from django.views import View
from asgiref.sync import async_to_sync
from aiohttp import BasicAuth

class BatchCall:
    def __init__(self, urls, bo_urls, username, password):
        self.urls = urls
        self.bo_urls = bo_urls
        self.auth = BasicAuth(username, password)

    async def fetch(self, session, url):
        try:
            async with session.get(url, auth=self.auth, ssl=False) as response:
                return await response.json()  # 根据需求返回对应格式
        except Exception as e:
            print(f"Problem: {url}: {e}")
            return None

    async def async_call_release(self):
        async with aiohttp.ClientSession() as session:
            tasks = [self.fetch(session, url) for url in self.urls]
            return await asyncio.gather(*tasks)

    async def async_call_bo(self):
        async with aiohttp.ClientSession() as session:
            tasks = [self.fetch(session, url) for url in self.bo_urls]
            return await asyncio.gather(*tasks)

# 在视图中使用
class BatchAPIView(View):
    def get(self, request):
        urls = ["url1", "url2"]  # 替换为你的URL列表
        bo_urls = ["bo_url1", "bo_url2"]
        batch_call = BatchCall(urls, bo_urls, "your_username", "your_password")
        
        # 同步调用异步方法
        results_release = async_to_sync(batch_call.async_call_release)()
        results_bo = async_to_sync(batch_call.async_call_bo)()
        
        return HttpResponse(f"Release Results: {results_release}<br>BO Results: {results_bo}")

验证方法 #

应用猴子补丁后，使用python manage.py runserver --noreload启动服务器，执行批量API调用，检查是否能完整执行所有100个请求且不退出。

内容的提问来源于stack exchange，提问作者xlxdxf