Django API服务突发OpenBLAS pthread_create失败问题求助

问题背景 #

• 运行一年的Django API服务，依赖Elasticsearch（间接依赖numpy），近期突然启动报错
• 本地及staging环境同代码逻辑可正常运行，仅生产环境出现异常
• 已尝试常规依赖重装、环境变量重置等方案，均无效

报错栈信息 #

File “/root/.cache/pypoetry/virtualenvs/REPLACED-wGPJvZ3g-py3.9/lib/python3.9/site-packages/elasticsearch/transport.py”, line 31, in <module>
from .serializer import DEFAULT_SERIALIZERS, Deserializer, JSONSerializer
File “/root/.cache/pypoetry/virtualenvs/REPLACED-wGPJvZ3g-py3.9/lib/python3.9/site-packages/elasticsearch/serializer.py”, line 35, in <module>
import numpy as np
File “/root/.cache/pypoetry/virtualenvs/REPLACED-wGPJvZ3g-py3.9/lib/python3.9/site-packages/numpy/__init__.py”, line 141, in <module>
from . import core
File “/root/.cache/pypoetry/virtualenvs/REPLACED-wGPJvZ3g-py3.9/lib/python3.9/site-packages/numpy/core/__init__.py”, line 23, in <module>
from . import multiarray
File “/root/.cache/pypoetry/virtualenvs/REPLACED-wGPJvZ3g-py3.9/lib/python3.9/site-packages/numpy/core/multiarray.py”, line 10, in <module>
from . import overrides
File “/root/.cache/pypoetry/virtualenvs/REPLACED-wGPJvZ3g-py3.9/lib/python3.9/site-packages/numpy/core/overrides.py”, line 6, in <module>
from numpy.core._multiarray_umath import (
KeyboardInterrupt
settings.INSTALLED_APPS
File “/root/.cache/pypoetry/virtualenvs/REPLACED-wGPJvZ3g-py3.9/lib/python3.9/site-packages/django/conf/__init__.py”, line 82, in __getattr__
self._setup(name)
File “/root/.cache/pypoetry/virtualenvs/REPLACED-wGPJvZ3g-py3.9/lib/python3.9/site-packages/django/conf/__init__.py”, line 69, in _setup
self._wrapped = Settings(settings_module)
File “/root/.cache/pypoetry/virtualenvs/REPLACED-wGPJvZ3g-py3.9/lib/python3.9/site-packages/django/conf/__init__.py”, line 170, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File “/usr/local/lib/python3.9/importlib/__init__.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “<frozen importlib._bootstrap>“, line 1030, in _gcd_import
File “<frozen importlib._bootstrap>“, line 1007, in _find_and_load
File “<frozen importlib._bootstrap>“, line 986, in _find_and_load_unlocked
File “<frozen importlib._bootstrap>“, line 680, in _load_unlocked
File “<frozen importlib._bootstrap_external>“, line 850, in exec_module
File “<frozen importlib._bootstrap>“, line 228, in _call_with_frames_removed
File “/usr/src/REPLACED/REPLACED/settings.py”, line 15, in <module>
from elasticsearch import Elasticsearch, RequestsHttpConnection
File “/root/.cache/pypoetry/virtualenvs/REPLACED-wGPJvZ3g-py3.9/lib/python3.9/site-packages/elasticsearch/__init__.py”, line 36, in <module>
from .client import Elasticsearch
File “/root/.cache/pypoetry/virtualenvs/REPLACED-wGPJvZ3g-py3.9/lib/python3.9/site-packages/elasticsearch/client/__init__.py”, line 23, in <module>
from ..transport import Transport, TransportError

OpenBLAS blas_thread_init: pthread_create failed for thread 1 of 4: Operation not permitted
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 2 of 4: Operation not permitted
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 3 of 4: Operation not permitted
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max

解决方案 #

• 限制OpenBLAS线程数：这是最直接的临时解决方案，避免多线程创建触发权限问题
  • 启动服务前设置环境变量：export OPENBLAS_NUM_THREADS=1
  • 或者在Django的settings.py最开头添加代码：

    import os
    os.environ['OPENBLAS_NUM_THREADS'] = '1'
    

• 调整系统进程数限制：检查当前用户的进程数上限，执行ulimit -u，若数值过低，修改/etc/security/limits.conf添加：

  root soft nproc 4096
  root hard nproc 8192
  

  保存后重新登录用户生效
• 替换OpenBLAS版本：使用单线程编译的OpenBLAS替代现有版本，或者通过pip安装不依赖OpenBLAS的numpy版本：pip install numpy --no-binary numpy
• 延迟依赖导入：将from elasticsearch import ...从settings.py移至视图函数、信号处理或其他延迟加载的模块中，避免在Django初始化阶段触发numpy/OpenBLAS的线程初始化

内容的提问来源于stack exchange，提问作者d3corator