Pytest执行失败：打开文件过多问题求助

问题详情 #

执行pytest -vvvv --setup-show tests命令时，测试用例收集阶段触发3个OSError: [Errno 24] Too many open files错误，涉及测试文件及目录。

错误日志 #

pytest -vvvv --setup-show tests
=========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.8.10, pytest-8.3.2, pluggy-1.5.0 -- <path>/.env/bin/python3
cachedir: .pytest_cache
rootdir: <path>
plugins: cov-5.0.0
collected 135 items / 3 errors

================================================================================================= ERRORS ==================================================================================================
________________________________________________________________________ ERROR collecting tests/functional/test_user_functional.py ________________________________________________________________________
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1014: in _gcd_import
    ???
<frozen importlib._bootstrap>:991: in _find_and_load
    ???
<frozen importlib._bootstrap>:975: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:671: in _load_unlocked
    ???
.env/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:165: in exec_module
    source_stat, co = _rewrite_test(fn, self.config)
.env/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:343: in _rewrite_test
    source = fn.read_bytes()
/usr/lib/python3.8/pathlib.py:1229: in read_bytes
    ???
/usr/lib/python3.8/pathlib.py:1222: in open
    ???
/usr/lib/python3.8/pathlib.py:1078: in _opener
    ???
E   OSError: [Errno 24] Too many open files: '<path>/functional/test_user_functional.py'
____________________________________________________________________________________ ERROR collecting tests/test_data _____________________________________________________________________________________
.env/lib/python3.8/site-packages/_pytest/runner.py:341: in from_call
    result: TResult | None = func()
.env/lib/python3.8/site-packages/_pytest/runner.py:389: in collect
    return list(collector.collect())
.env/lib/python3.8/site-packages/_pytest/main.py:523: in collect
    for direntry in scandir(self.path):
.env/lib/python3.8/site-packages/_pytest/pathlib.py:885: in scandir
    with os.scandir(path) as s:
E   OSError: [Errno 24] Too many open files: '<path>/test_data'
_______________________________________________________________________________________ ERROR collecting tests/unit _______________________________________________________________________________________
.env/lib/python3.8/site-packages/_pytest/runner.py:341: in from_call
    result: TResult | None = func()
.env/lib/python3.8/site-packages/_pytest/runner.py:389: in collect
    return list(collector.collect())
.env/lib/python3.8/site-packages/_pytest/main.py:523: in collect
    for direntry in scandir(self.path):
.env/lib/python3.8/site-packages/_pytest/pathlib.py:885: in scandir
    with os.scandir(path) as s:
E   OSError: [Errno 24] Too many open files: '<path>/unit'
============================================================================================ warnings summary =============================================================================================
.env/lib/python3.8/site-packages/pydantic/_internal/_fields.py:201
  <path>/.env/lib/python3.8/site-packages/pydantic/_internal/_fields.py:201: UserWarning: Field name "schema" in "SnowflakeConnectModel" shadows an attribute in parent "BaseModel"

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================================================= short test summary info =========================================================================================
ERROR tests/functional/test_user_functional.py - OSError: [Errno 24] Too many open files: '<path>/functional/test_user_functional.py'
ERROR tests/test_data - OSError: [Errno 24] Too many open files: '<path>/test_data'
ERROR tests/unit - OSError: [Errno 24] Too many open files: '<path>/unit'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 3 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================================================================== 1 warning, 3 errors in 3.06s =======================================================================================

当前系统文件句柄限制 #

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 29053
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 29053
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

解决方法 #

1. 临时提升当前会话文件句柄限制 #

执行以下命令临时调高允许打开的文件数（示例设置为4096）：

ulimit -n 4096

设置完成后重新运行pytest即可，注意该设置仅在当前终端会话有效，关闭终端后恢复默认值。

2. 永久提升文件句柄限制 #

若需永久生效，修改系统配置：

• 编辑/etc/security/limits.conf，添加或修改以下内容：

  * soft nofile 4096
  * hard nofile 8192
  

  其中soft为警告阈值，hard为最大限制值。
• 编辑/etc/pam.d/common-session，确保存在该行：

  session required pam_limits.so
  

• 重启系统或重新登录后，运行ulimit -n确认设置已生效。

3. 排查代码中的文件句柄泄漏 #

若调高限制后问题依旧，需检查代码或依赖库是否存在资源未释放的情况：

• 检查测试代码中是否存在打开文件后未调用close()，或未使用with语句管理文件资源的情况。
• 使用lsof -p <pytest进程PID>命令查看进程打开的文件列表，定位泄漏源。
• 排查第三方库是否在初始化时打开大量文件未释放。

4. 优化pytest运行方式 #

• 分模块运行测试，避免一次性加载所有文件：比如单独执行pytest tests/functional/，再运行单元测试。
• 禁用不必要的插件，如无需覆盖率统计可去掉--cov参数，减少插件带来的额外文件操作。

内容的提问来源于stack exchange，提问作者Moravas