YOLOv5训练报WinError 1455分页文件过小错误该如何解决？

问题场景 #

在包含约10000个训练样本的自定义数据集上训练YOLOv5卷积神经网络时触发OSError报错，报错前程序已占满30GB分页文件+10GB已分配RAM。

报错日志 #

C:\Program Files\Python39\lib\site-packages\setuptools\distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
  warnings.warn(
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Program Files\Python39\lib\multiprocessing\spawn.py", line 125, in _main
    prepare(preparation_data)
  File "C:\Program Files\Python39\lib\multiprocessing\spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Program Files\Python39\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "C:\Program Files\Python39\lib\runpy.py", line 268, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "C:\Program Files\Python39\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "C:\Program Files\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Malth\OneDrive - Aarhus Universitet\7. Semester\DL\YOLO\yolov5\train.py", line 20, in <module>
    import torch
  File "C:\Users\Malth\AppData\Roaming\Python\Python39\site-packages\torch\__init__.py", line 124, in <module>
    raise err
OSError: [WinError 1455] The paging file is too small for this operation to complete. Error loading "C:\Users\Malth\AppData\Roaming\Python\Python39\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll" or one of its dependencies.

运行命令 #

python train.py --rect --batch 16 --epochs 3 --data CCPDMini.yaml --weights yolov5s.pt

环境信息 #

• Python版本：3.9.2
• PyTorch版本：1.9.1+cu111
• 硬件配置：Intel Core i7-4790 CPU、16 GB RAM、RTX2070 8GB VRAM

已尝试无效方案 #

• 降低batch size
• 减少dataloader的工作进程数

可行解决方案 #

• 调整Windows虚拟内存分页文件大小：将分页文件最大值调整为物理内存的3_{4倍，也就是48GB}64GB，设置完成后重启电脑生效。
• 释放系统内存：训练开始前关闭浏览器、办公软件等占用内存的后台程序，尽可能预留更多物理内存给训练进程。
• 调整训练参数：
  • 在运行命令中添加--workers 0，直接禁用多进程数据加载，避免多进程带来的额外内存开销
  • 进一步将batch size下调到4或2，同时可选择添加--cache disk参数，将数据集缓存从内存改为磁盘缓存，大幅降低内存占用
• 升级PyTorch版本：当前使用的1.9.1版本属于较老的CUDA适配版本，存在部分内存泄漏的已知问题，可升级到适配CUDA11.x的更高稳定版PyTorch，减少不必要的内存占用。
• 校验数据集：确认所有标注文件格式符合YOLOv5要求，不存在无效标注、超大尺寸标注等会导致训练时内存异常占用的问题。

内容的提问来源于stack exchange，提问作者MaltDew