调用OpenEdx /api/courseware/sequence接口偶发500错误求助

问题背景 #

• 部署环境：Ubuntu服务器通过Tutor部署OpenEdx，同服务器Docker运行独立Django项目，该项目频繁调用OpenEdx的/api/courseware/sequence/{{block_key}}接口
• 现象：多数请求返回200正常响应，偶发返回500内部服务器错误（非必现）
• 核心异常：从LMS日志中捕获到django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

完整错误日志 #

lms_1                        | 2023-01-02 18:32:42,511 ERROR 19 [django.request] [user 42] [ip 172.18.0.14] log.py:224 - Internal Server Error: /api/courseware/sequence/block-v1:SOS+CSE-10+2022_r1+type@sequential+block@b9aad4acdd1f48259c3dfd92fdc216c2
lms_1                        | Traceback (most recent call last):
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
lms_1                        |     response = get_response(request)
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
lms_1                        |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
lms_1                        |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/contextlib.py", line 75, in inner
lms_1                        |     return func(*args, **kwds)
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
lms_1                        |     return view_func(*args, **kwargs)
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
lms_1                        |     return self.dispatch(request, *args, **kwargs)
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
lms_1                        |     response = self.handle_exception(exc)
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
lms_1                        |     response = handler(request, *args, **kwargs)
lms_1                        |   File "/openedx/edx-platform/./openedx/core/djangoapps/courseware_api/views.py", line 596, in get
lms_1                        |     sequence, _ = get_module_by_usage_id(
lms_1                        |   File "/openedx/edx-platform/./lms/djangoapps/courseware/module_render.py", line 1070, in get_module_by_usage_id
lms_1                        |     instance = get_module_for_descriptor(
lms_1                        |   File "/openedx/edx-platform/./lms/djangoapps/courseware/module_render.py", line 387, in get_module_for_descriptor
lms_1                        |     return get_module_for_descriptor_internal(
lms_1                        |   File "/openedx/edx-platform/./lms/djangoapps/courseware/module_render.py", line 806, in get_module_for_descriptor_internal
lms_1                        |     (system, student_data) = get_module_system_for_user(
lms_1                        |   File "/openedx/edx-platform/./lms/djangoapps/courseware/module_render.py", line 724, in get_module_system_for_user
lms_1                        |     field_data = DateLookupFieldData(descriptor._field_data, course_id, user)  # pylint: disable=protected-access
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/edx_when/field_data.py", line 51, in __init__
lms_1                        |     self._load_dates(course_id, user, use_cached=use_cached)
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/edx_when/field_data.py", line 59, in _load_dates
lms_1                        |     for (location, field), date in api.get_dates_for_course(course_id, user, use_cached=use_cached).items():
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/edx_when/api.py", line 186, in get_dates_for_course
lms_1                        |     allow_relative_dates = _are_relative_dates_enabled(course_id)
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/edx_when/api.py", line 70, in _are_relative_dates_enabled
lms_1                        |     return RELATIVE_DATES_FLAG.is_enabled(course_key)
lms_1                        |   File "/openedx/edx-platform/./openedx/core/djangoapps/waffle_utils/__init__.py", line 122, in is_enabled
lms_1                        |     is_enabled_for_course = self._get_course_override_value(course_key)
lms_1                        |   File "/openedx/edx-platform/./openedx/core/djangoapps/waffle_utils/__init__.py", line 78, in _get_course_override_value
lms_1                        |     course_override = WaffleFlagCourseOverrideModel.override_value(
lms_1                        |   File "/openedx/edx-platform/./openedx/core/lib/cache_utils.py", line 74, in decorator
lms_1                        |     result = wrapped(*args, **kwargs)
lms_1                        |   File "/openedx/edx-platform/./openedx/core/djangoapps/waffle_utils/models.py", line 60, in override_value
lms_1                        |     effective = cls.objects.filter(waffle_flag=waffle_flag, course_id=course_id).order_by('-change_date').first()
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/db/models/query.py", line 674, in first
lms_1                        |     for obj in (self if self.ordered else self.order_by('pk'))[:1]:
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/db/models/query.py", line 280, in __iter__
lms_1                        |     self._fetch_all()
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1324, in _fetch_all
lms_1                        |     self._result_cache = list(self._iterable_class(self))
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/db/models/query.py", line 51, in __iter__
lms_1                        |     results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
lms_1                        |     cursor.execute(sql, params)
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
lms_1                        |     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
lms_1                        |     return executor(sql, params, many, context)
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 78, in _execute
lms_1                        |     self.db.validate_no_broken_transaction()
lms_1                        |   File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 447, in validate_no_broken_transaction
lms_1                        |     raise TransactionManagementError(
lms_1                        | django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.
lms_1                        | [pid: 19|app: 0|req: 28631/59539] 172.18.0.6 () {40 vars in 950 bytes} [Mon Jan  2 18:32:42 2023] GET /api/courseware/sequence/block-v1:SOS+CSE-10+2022_r1+type@sequential+block@b9aad4acdd1f48259c3dfd92fdc216c2 => generated 9585 bytes in 317 msecs (HTTP/1.1 500) 7 headers in 507 bytes (1 switches on core 0)

排查分析 #

1. 异常本质：代码在一个已经失败的atomic事务块内尝试执行新的数据库查询，违反Django事务管理规则——当事务因之前的操作（如锁冲突、连接超时）进入broken状态后，必须先回滚或结束当前事务，才能执行新查询。
2. 触发路径：请求进入courseware_api的sequence接口后，加载课程日期配置时调用RELATIVE_DATES_FLAG.is_enabled()检查课程级开关，该方法内部查询WaffleFlagCourseOverrideModel时，当前请求的事务已处于异常状态。
3. 偶发原因：只有当之前的数据库操作出现隐性失败且未被及时捕获处理时，后续查询才会触发该错误，因此是非必现的。

解决方法 #

临时缓解方案 #

• 添加重试机制：在Django调用端对500错误实现自动重试（最多3次，间隔1-2秒），避开事务异常的请求上下文。
• 降低请求频率：减少Django项目对该接口的并发调用次数，降低事务冲突概率。

永久修复方案（OpenEdx端） #

1. 修复事务异常处理：在edx_when/api.py的_are_relative_dates_enabled方法或上层逻辑中，添加事务状态检查与异常处理，确保在查询前重置异常事务：

   from django.db import transaction, DatabaseError
   
   def _are_relative_dates_enabled(course_id):
       try:
           conn = transaction.get_connection()
           # 检查当前事务是否处于异常状态，若则回滚
           if conn.in_atomic_block and conn.needs_rollback:
               transaction.rollback()
           return RELATIVE_DATES_FLAG.is_enabled(course_id)
       except DatabaseError:
           # 捕获异常，默认返回False或按业务逻辑处理
           return False
   

2. 优化缓存策略：确认WaffleFlagCourseOverrideModel.override_value的缓存配置生效，设置合理的缓存过期时间（如10-15分钟），减少重复数据库查询。
3. 升级OpenEdx版本：该问题可能是已知bug，检查官方补丁后升级到包含事务处理修复的稳定版本（如Ironwood及之后版本）。

内容的提问来源于stack exchange，提问作者Amit Kumar