如何解决PyTorch训练时的RuntimeError文件写入失败问题

我在已正常运行数小时的StyleGAN2模型训练会话中遇到如下错误：
RuntimeError: [enforce fail at inline_container.cc:588] PytorchStreamWriter failed writing file data/17: file write failed

完整堆栈跟踪信息如下：

Traceback (most recent call last):
  File "/opt/conda/bin/stylegan2_pytorch", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/stylegan2_pytorch/cli.py", line 190, in main
    fire.Fire(train_from_folder)
  File "/opt/conda/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/conda/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/opt/conda/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/stylegan2_pytorch/cli.py", line 184, in train_from_folder
    mp.spawn(run_training,
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 246, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 202, in start_processes
    while not context.join():
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 163, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 619, in save
    _save(obj, opened_zipfile, pickle_module, pickle_protocol, _disable_byteorder_record)
  File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 853, in _save
    zip_file.write_record(name, storage.data_ptr(), num_bytes)
RuntimeError: [enforce fail at inline_container.cc:588] . PytorchStreamWriter failed writing file data/17: file write failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 74, in _wrap
    fn(i, *args)
  File "/opt/conda/lib/python3.10/site-packages/stylegan2_pytorch/cli.py", line 60, in run_training
    retry_call(model.train, tries=3, exceptions=NanException)
  File "/opt/conda/lib/python3.10/site-packages/retry/api.py", line 101, in retry_call
    return __retry_internal(partial(f, *args, **kwargs), exceptions, tries, delay, max_delay, backoff, jitter, logger)
  File "/opt/conda/lib/python3.10/site-packages/retry/api.py", line 33, in __retry_internal
    return f()
  File "/opt/conda/lib/python3.10/site-packages/stylegan2_pytorch/stylegan2_pytorch.py", line 1147, in train
    self.save(self.checkpoint_num)
  File "/opt/conda/lib/python3.10/site-packages/stylegan2_pytorch/stylegan2_pytorch.py", line 1368, in save
    torch.save(save_data, self.model_name(num))
  File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 618, in save
    with _open_zipfile_writer(f) as opened_zipfile:
  File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 466, in __exit__
    self.file_like.write_end_of_file()
RuntimeError: [enforce fail at inline_container.cc:424] . unexpected pos 79331008 vs 79330896

/opt/conda/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

最后保存的文件大小与之前的不同，删除该文件后重启训练，错误仍然会重现，恳请提供解决帮助。

解决建议 #

• 检查磁盘空间与权限：执行df -h查看磁盘剩余空间，确认是否已满；用ls -l检查模型保存目录的读写权限，确保当前用户有写入权限。
• 更换保存路径：如果当前路径是远程挂载存储或存在文件系统问题，尝试将模型保存到本地磁盘的其他目录，修改训练脚本中的保存路径配置。
• 调整多进程保存逻辑：从堆栈看出训练使用了多进程，多进程下同时写入模型文件可能引发竞争。可以改为单进程训练，或者修改代码让主进程单独负责模型保存，避免子进程参与写入。
• 降低保存频率：临时减少模型保存的次数，比如从每1000步保存改为每2000步保存，排查是否是大模型写入时的IO超时问题。
• 升级PyTorch版本：旧版本PyTorch的torch.save存在ZIP容器写入的bug，尝试升级到1.13.x或2.x的稳定版本，修复底层写入问题。
• 检测存储硬件状态：如果是本地磁盘，用smartctl等工具检查是否存在坏道；如果是云存储，确认服务是否有临时故障。

内容的提问来源于stack exchange，提问作者jeremy_rutman