Celery Eventlet环境下OSError：无法获取系统负载均值问题排查与预防

我们的Django Celery服务器采用Eventlet作为执行池，并发值设置为500，近期首次遇到OSError: Load averages are unobtainable错误，重启服务器后问题暂时解决，现寻求该问题的永久预防方案。

Python包版本 #

• Django 4.1.7
• celery 5.2.3
• eventlet 0.33.3

系统文件打开限制 #

软限制 #

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31071
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 2048
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31071
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

硬限制 #

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31071
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 16384
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31071
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

错误回溯信息 #

Traceback (most recent call last):
 File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
  timer()
 File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/eventlet/hubs/timer.py", line 59, in __call__
  cb(*args, **kw)
 File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 224, in main
  self._resolve_links()
 File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 240, in _resolve_links
  f(self, *ca, **ckw)
 File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/celery/concurrency/eventlet.py", line 62, in _entry_exit
  g.wait()
 File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 181, in wait
  return self._exit_event.wait()
 File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/eventlet/event.py", line 132, in wait
  current.throw(*self._exc)
 File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 221, in main
  result = function(*args, **kwargs)
 File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/kombu/asynchronous/timer.py", line 70, in __call__
  return self.fun(*self.args, **self.kwargs)
 File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/kombu/asynchronous/timer.py", line 137, in _reschedules
  return fun(*args, **kwargs)
 File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/celery/worker/heartbeat.py", line 45, in _send
  loadavg=load_average(),
 File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/celery/utils/sysinfo.py", line 23, in load_average
  return _load_average()
 File "/home/ubuntu/project_data/venv/lib/python3.10/site-packages/celery/utils/sysinfo.py", line 13, in _load_average
  return tuple(ceil(l * 1e2) / 1e2 for l in os.getloadavg())
OSError: Load averages are unobtainable

永久预防方案 #

1. 提升文件打开软限制 #

当前软限制为2048，远低于硬限制16384。高并发场景下，Eventlet协程会占用大量文件句柄，容易触发资源耗尽问题，导致系统调用os.getloadavg()失败。

• 临时生效：执行命令 ulimit -n 16384
• 永久生效：编辑/etc/security/limits.conf，添加以下内容（替换your_username为运行Celery的用户）：

  your_username soft nofile 16384
  your_username hard nofile 16384
  

  保存后重启系统或重新登录用户。

2. 合理调整Celery并发数 #

500的并发数可能超出当前服务器的负载能力，建议逐步降低并发值（比如先调整到300），观察错误是否复现。Eventlet的并发数需结合服务器CPU、内存、IO资源实际情况配置，并非越高越好。

3. 禁用Celery心跳的负载上报 #

错误触发点在Celery心跳发送时获取系统负载，可直接禁用该功能避免报错：

• 在Django配置文件中添加：

  CELERY_WORKER_HEARTBEAT = False
  

• 或启动worker时添加参数：

  celery worker --without-heartbeat
  

4. 更新依赖包到稳定版 #

当前celery 5.2.3、eventlet 0.33.3版本较旧，新版本可能修复了系统负载获取的相关bug，执行以下命令更新：

pip install --upgrade celery eventlet

5. 监控系统资源使用 #

定期用top、htop、lsof工具监控服务器的CPU、内存、文件句柄使用情况：

• 查看特定worker进程的文件句柄数量：lsof -p <worker_pid> | wc -l
• 实时监控系统负载：watch -n 1 cat /proc/loadavg
  通过监控及时发现资源瓶颈，提前调整配置。

内容的提问来源于stack exchange，提问作者Shravan Nani