容器运行Django+Celery访问MySQL报(2000, 'Unknown MySQL error')

我在Django应用中集成使用Celery，应用本身访问MySQL数据库运行正常，但当Celery尝试访问数据库时会抛出(2000, 'Unknown MySQL error')错误。该问题仅在容器中运行Celery时出现，在Ubuntu宿主机直接运行Celery时功能完全正常。

获取到的完整报错信息如下：

[2022-06-18 13:39:33,717: ERROR/ForkPoolWorker-1] Task taskMonitor.tasks.monitor[7e6696aa-d602-4336-a582-4c719f8d72df] raised unexpected: OperationalError(2000, 'Unknown MySQL error')
Traceback (most recent call last):
  File "/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/.venv/lib/python3.9/site-packages/django/db/backends/mysql/base.py", line 75, in execute
    return self.cursor.execute(query, args)
  File "/.venv/lib/python3.9/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/.venv/lib/python3.9/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/.venv/lib/python3.9/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (2000, 'Unknown MySQL error')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/.venv/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/.venv/lib/python3.9/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/app/taskMonitor/tasks.py", line 18, in monitor
    for obj in objs.iterator():
  File "/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 401, in _iterator
    yield from self._iterable_class(
  File "/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 57, in __iter__
    results = compiler.execute_sql(
  File "/.venv/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1361, in execute_sql
    cursor.execute(sql, params)
  File "/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 103, in execute
    return super().execute(sql, params)
  File "/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/.venv/lib/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/.venv/lib/python3.9/site-packages/django/db/backends/mysql/base.py", line 75, in execute
    return self.cursor.execute(query, args)
  File "/.venv/lib/python3.9/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/.venv/lib/python3.9/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/.venv/lib/python3.9/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (2000, 'Unknown MySQL error')

这个错误是Celery默认prefork工作池的典型兼容问题：Celery主进程启动时会初始化Django环境，建立MySQL连接，fork出worker子进程时，子进程会直接继承父进程的数据库连接。而MySQL连接本身不支持跨进程安全复用，容器环境默认的进程安全策略会直接拦截这种跨进程复用的连接操作，最终抛出无明确提示的2000错误。宿主机运行时没有对应的安全拦截规则，所以不会触发该问题。

按优先级从高到低尝试以下操作：

• 配置Celery子进程初始化时主动关闭旧连接，在Celery配置文件中添加如下信号处理逻辑：

from celery.signals import worker_process_init
from django.db import close_old_connections

@worker_process_init.connect
def clean_db_connection(**kwargs):
    close_old_connections()

• 调整Celery工作池模式，先加--pool=solo参数启动worker验证问题是否复现，确认是fork导致的连接问题后，可以切换为gevent池规避fork带来的连接继承问题，启动命令参考：
  celery -A 你的项目名 worker -l info --pool=gevent --concurrency=4
• 检查容器内MySQL客户端依赖兼容性，不要使用alpine基础镜像的精简MySQL客户端包，换成Debian基础镜像安装完整版libmysqlclient-dev依赖后，再重新编译安装mysqlclient库，避免客户端版本和服务端不匹配导致的未知错误。
• 修改Django数据库配置禁用长连接，在DATABASES配置项中添加CONN_MAX_AGE=0参数，强制每次数据库操作前检查连接有效性，失效则自动重建连接，彻底避免旧连接跨进程复用的问题。

注意：不要在Celery任务装饰器外部提前构造QuerySet对象，这类操作会在主进程初始化阶段就触发数据库连接建立，所有数据库查询逻辑都要放在任务函数内部执行。

内容的提问来源于stack exchange，提问作者michael