JupyterLab报'Too many open files'错误：库重复加载问题排查

数据老陈

2026-6-27

问题背景

反复遇到JupyterLab会话挂起，终端持续输出回溯信息，最终报错：

zmq.error.ZMQError: Too many open files

完整回溯信息：

[E 2024-03-17 17:40:36.843 ServerApp] Uncaught exception GET /api/kernels/059a770b-436f-40b5-b41b-95a1748ef7fd/channels?session_id=f60b1486-f926-4c96-924d-692a4e50d1e4 (172.16.206.48)
    HTTPServerRequest(protocol='http', host='localhost:23456', method='GET', uri='/api/kernels/059a770b-436f-40b5-b41b-95a1748ef7fd/channels?session_id=f60b1486-f926-4c96-924d-692a4e50d1e4', version='HTTP/1.1', remote_ip='172.16.206.48')
    Traceback (most recent call last):
      File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/tornado/websocket.py", line 944, in _accept_connection
        await open_result
      File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/jupyter_server/services/kernels/websocket.py", line 77, in open
        await self.connection.connect()
      File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/jupyter_server/services/kernels/connection/channels.py", line 363, in connect
        self.create_stream()
      File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/jupyter_server/services/kernels/connection/channels.py", line 154, in create_stream
        self.channels[channel] = stream = meth(identity=identity)
      File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/jupyter_client/ioloop/manager.py", line 25, in wrapped
        socket = f(self, *args, **kwargs)
      File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/jupyter_client/connect.py", line 664, in connect_iopub
        sock = self._create_connected_socket("iopub", identity=identity)
      File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/jupyter_client/connect.py", line 654, in _create_connected_socket
        sock = self.context.socket(socket_type)
      File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/zmq/sugar/context.py", line 362, in socket
        s: ST = socket_class(  # set PYTHONTRACEMALLOC=2 to get the calling frame
      File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/zmq/sugar/socket.py", line 159, in __init__
        super().__init__(
      File "zmq/backend/cython/socket.pyx", line 332, in zmq.backend.cython.socket.Socket.__init__
    zmq.error.ZMQError: Too many open files

会话仅运行单个笔记本，规模较大但未达极端。通过lsof排查发现多个库文件被重复计数30余次：

$ lsof | grep scipy/sparse/_sparsetools.cpython-310-x86_64-linux-gnu.so | wc
     33     361    8613
$ lsof | grep indexing.cpython-310-x86_64-linux-gnu.so | wc
     33     361    8481

具体文件打开信息显示，这些库由同一PID进程下的不同TID线程关联：

COMMAND       PID     TID TASKCMD             USER   FD      TYPE             DEVICE    SIZE/OFF               NODE NAME
python    1325008                     adamginsburg  mem       REG        2445,764964     4384216 180149904096755221 /blue/adamginsburg/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/scipy/sparse/_sparsetools.cpython-310-x86_64-linux-gnu.so
python    1325008 1325015 ZMQbg/Rea   adamginsburg  mem       REG        2445,764964     4384216 180149904096755221 /blue/adamginsburg/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/scipy/sparse/_sparsetools.cpython-310-x86_64-linux-gnu.so
python    1325008 1325016 ZMQbg/IO/   adamginsburg  mem       REG        2445,764964     4384216 180149904096755221 /blue/adamginsburg/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/scipy/sparse/_sparsetools.cpython-310-x86_64-linux-gnu.so
python    1325008 1325017 python      adamginsburg  mem       REG        2445,764964     4384216 180149904096755221 /blue/adamginsburg/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/scipy/sparse/_sparsetools.cpython-310-x86_64-linux-gnu.so
python    1325008 1325018 python      adamginsburg  mem       REG        2445,764964     4384216 180149904096755221 /blue/adamginsburg/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/scipy/sparse/_sparsetools.cpython-310-x86_64-linux-gnu.so

原因分析

lsof计数的本质：所谓"重复打开"并非真的重复加载库文件。Linux中每个线程会为已加载的共享库创建独立的内存映射条目，lsof会将这些条目分别计数，但库文件实际仅加载一次到内存。不过这些条目仍会占用文件句柄配额，线程过多时就会触发句柄耗尽错误。
线程累积问题：从TID的任务名（ZMQbg/Rea、ZMQbg/IO/）能看出，ZMQ后台线程在持续创建，再加上笔记本代码中可能的多线程操作（如并行计算、异步任务），导致进程内线程数量不断增长，每个线程关联的共享库映射条目累积，最终耗尽文件句柄。
线程泄漏隐患：Python线程如果未被正确回收（比如未调用join()、线程池未关闭），会持续占用资源，关联的共享库映射条目也不会释放。

解决方法

1. 提升系统文件句柄限制

临时调整当前会话的句柄上限：

# 查看当前限制
ulimit -n
# 临时设置更高值（如65535）
ulimit -n 65535

永久修改需编辑/etc/security/limits.conf，添加：
```
adamginsburg soft nofile 65535
adamginsburg hard nofile 65535
```
修改后需重新登录生效。

2. 控制线程数量

检查笔记本代码中的多线程逻辑，改用线程池替代手动创建大量线程，限制线程池大小（如concurrent.futures.ThreadPoolExecutor(max_workers=8)）。
减少ZMQ后台线程：生成并编辑Jupyter配置文件jupyter_server_config.py，添加：
```
c.KernelManager.io_threads = 2
```

3. 优化线程回收与库加载

确保线程任务完成后被正确回收，比如显式调用thread.join()，或用上下文管理器管理线程池。
在主线程提前加载频繁使用的C扩展库，减少线程触发的映射条目创建。

4. 升级依赖修复bug

更新Jupyter相关包和ZMQ库，修复可能存在的线程泄漏或句柄管理问题：

pip install --upgrade jupyterlab jupyter_server pyzmq scipy

内容的提问来源于stack exchange，提问作者keflavich

JupyterLab报'Too many open files'错误：库重复加载问题排查

原因分析 #

解决方法 #

1. 提升系统文件句柄限制 #

2. 控制线程数量 #

3. 优化线程回收与库加载 #