刷新Django API请求时触发Cannot use MongoClient异常求助

问题描述 #

• 在AWS EC2实例上通过Apache（httpd）部署基于Django、Djongo、Django Rest Framework的Web服务
• 首次API请求可正常返回JSON响应，再次调用时触发DatabaseError，服务表现为有时正常、有时异常
• 排查后发现核心异常为：Cannot use MongoClient after close

完整异常栈 #

DatabaseError at /profile/7/
No exception message supplied

Environment:

Request Method: GET
Request URL: https://[redacted]/profile/7/?format=json

Django Version: 3.2.16
Python Version: 3.7.10
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'rest_framework',
 'api']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/djongo/sql2mongo/query.py", line 808, in __iter__
    yield from iter(self._query)
  File "/usr/local/lib/python3.7/site-packages/djongo/sql2mongo/query.py", line 166, in __iter__
    for doc in cursor:
  File "/usr/local/lib64/python3.7/site-packages/pymongo/cursor.py", line 1248, in next
    if len(self.__data) or self._refresh():
  File "/usr/local/lib64/python3.7/site-packages/pymongo/cursor.py", line 1165, in _refresh
    self.__send_message(q)
  File "/usr/local/lib64/python3.7/site-packages/pymongo/cursor.py", line 1053, in __send_message
    operation, self._unpack_response, address=self.__address
  File "/usr/local/lib64/python3.7/site-packages/pymongo/_csot.py", line 105, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib64/python3.7/site-packages/pymongo/mongo_client.py", line 1335, in _run_operation
    retryable=isinstance(operation, message._Query),
  File "/usr/local/lib64/python3.7/site-packages/pymongo/_csot.py", line 105, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib64/python3.7/site-packages/pymongo/mongo_client.py", line 1441, in _retryable_read
    server = self._select_server(read_pref, session, address=address)
  File "/usr/local/lib64/python3.7/site-packages/pymongo/mongo_client.py", line 1247, in _select_server
    topology = self._get_topology()
  File "/usr/local/lib64/python3.7/site-packages/pymongo/mongo_client.py", line 1203, in _get_topology
    self._topology.open()
  File "/usr/local/lib64/python3.7/site-packages/pymongo/topology.py", line 192, in open
    self._ensure_opened()
  File "/usr/local/lib64/python3.7/site-packages/pymongo/topology.py", line 592, in _ensure_opened
    raise InvalidOperation("Cannot use MongoClient after close")

The above exception (Cannot use MongoClient after close) was the direct cause of the following exception:
  File "/usr/local/lib/python3.7/site-packages/djongo/cursor.py", line 65, in fetchmany
    ret.append(self.result.next())
  File "/usr/local/lib/python3.7/site-packages/djongo/sql2mongo/query.py", line 797, in __next__
    result = next(self._result_generator)
  File "/usr/local/lib/python3.7/site-packages/djongo/sql2mongo/query.py", line 830, in __iter__
    raise exe from e

The above exception (
    Keyword: FAILED SQL: SELECT "api_account"."userID", "api_account"."phoneNum", "api_account"."displayName", "api_account"."seeking", "api_account"."birthday", "api_account"."motto", "api_account"."faithID", "api_account"."job", "api_account"."hobbies", "api_account"."politicsID", "api_account"."city", "api_account"."stateID", "api_account"."bio", "api_account"."facePlot" FROM "api_account" WHERE "api_account"."userID" = %(0)s LIMIT 21
Params: (7,)
Version: 1.3.6
    Sub SQL: None
    FAILED SQL: None
    Params: None
    Version: None) was the direct cause of the following exception:
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 97, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/djongo/cursor.py", line 70, in fetchmany
    raise db_exe from e

The above exception () was the direct cause of the following exception:
  File "/usr/local/lib/python3.7/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/rest_framework/generics.py", line 282, in get
    return self.retrieve(request, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/rest_framework/mixins.py", line 54, in retrieve
    instance = self.get_object()
  File "/usr/local/lib/python3.7/site-packages/rest_framework/generics.py", line 96, in get_object
    obj = get_object_or_404(queryset, **filter_kwargs)
  File "/usr/local/lib/python3.7/site-packages/rest_framework/generics.py", line 19, in get_object_or_404
    return _get_object_or_404(queryset, *filter_args, **filter_kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/shortcuts.py", line 76, in get_object_or_404
    return queryset.get(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 431, in get
    num = len(clone)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 51, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1208, in execute_sql
    return list(result)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1646, in cursor_iter
    for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel):
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1646, in <lambda>
    for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel):
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 97, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 97, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/djongo/cursor.py", line 70, in fetchmany
    raise db_exe from e

Exception Type: DatabaseError at /profile/7/
Exception Value: 

使用的Python库版本 #

• Django: 3.2.16
• djangorestframework: 3.14.0
• djongo: 1.3.6
• pymongo: 4.3.2
• sqlparse: 0.2.4

• 版本兼容性问题：Djongo 1.3.6是较旧版本，发布时仅适配pymongo 3.x系列，而pymongo 4.3.2在MongoClient的连接管理、生命周期处理上有较大改动。Djongo的连接池逻辑未适配pymongo 4.x的连接关闭机制，导致连接被关闭后，后续请求尝试复用已失效的MongoClient实例。
• Apache多进程/线程环境的连接冲突：Apache通过mod_wsgi部署Django时，会启动多个工作进程或线程。Djongo的连接池未正确实现多进程环境下的连接隔离，某个进程关闭数据库连接后，其他进程可能尝试复用这个已关闭的连接实例，触发Cannot use MongoClient after close错误。
• Django连接生命周期适配问题：Djongo作为Django的数据库后端，未严格遵循Django的数据库连接生命周期规范。正常情况下Django会在请求结束后关闭连接，新请求到来时重新创建连接，但Djongo可能错误地复用了已关闭的连接，导致请求失败。

内容的提问来源于stack exchange，提问作者user3864563