PyTorch DataLoader多进程训练报错：Bad file descriptor与EOFError求助

问题描述 #

在使用自定义数据集构建的PyTorch DataLoader进行神经网络训练时遇到异常：设置num_workers=4、pin_memory=False时，训练多数情况可正常完成，但会随机在某一阶段停止，抛出两类错误：

• OSError: [Errno 9] Bad file descriptor
• EOFError
  该错误仅在集群环境中出现，将workers设为0时错误消失，但需要多进程加速训练。

环境版本 #

• Python 3.9.12
• PyTorch 1.11.0+cu102

错误栈信息 #

Traceback (most recent call last):
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/resource_sharer.py", line 145, in _serve
Epoch 17:  52%|█████▏    | 253/486 [01:00<00:55,  4.18it/s, loss=1.73]

Traceback (most recent call last):
  File "/my_directory/bench/run_experiments.py", line 251, in <module>
    send(conn, destination_pid)
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/resource_sharer.py", line 50, in send
    reduction.send_handle(conn, new_fd, pid)
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/reduction.py", line 183, in send_handle
    with socket.fromfd(conn.fileno(), socket.AF_UNIX, socket.SOCK_STREAM) as s:
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/socket.py", line 545, in fromfd
    return socket(family, type, proto, nfd)
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/socket.py", line 232, in __init__
    _socket.socket.__init__(self, family, type, proto, fileno)
OSError: [Errno 9] Bad file descriptor

    main(args)
  File "/my_directory/bench/run_experiments.py", line 183, in main
    run_experiments(args, save_path)
  File "/my_directory/bench/run_experiments.py", line 70, in run_experiments
    ) = run_algorithm(algorithm_params[j], mp[j], ss, dataset)
  File "/my_directorybench/algorithms.py", line 38, in run_algorithm
    data = es(mp,search_space,  dataset, **ps)
  File "/my_directorybench/algorithms.py", line 151, in es
   data = ss.generate_random_dataset(mp,
  File "/my_directorybench/architectures.py", line 241, in generate_random_dataset
    arch_dict = self.query_arch(
  File "/my_directory/bench/architectures.py", line 71, in query_arch
    train_losses, val_losses, model = meta_net.get_val_loss(
  File "/my_directory/bench/meta_neural_net.py", line 50, in get_val_loss
    return self.training(
  File "/my_directorybench/meta_neural_net.py", line 155, in training
    train_loss = self.train_step(model, device, train_loader, epoch)
  File "/my_directory/bench/meta_neural_net.py", line 179, in train_step
    for batch_idx, mini_batch in enumerate(pbar):
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/my_directory/.local/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 530, in __next__
    data = self._next_data()
  File "/my_directory/.local/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1207, in _next_data
    idx, data = self._get_data()
  File "/my_directory/.local/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1173, in _get_data
    success, data = self._try_get_data()
  File "/my_directory/.local/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1011, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/queues.py", line 122, in get
    return _ForkingPickler.loads(res)
  File "/my_directory/.local/lib/python3.9/site-packages/torch/multiprocessing/reductions.py", line 295, in rebuild_storage_fd
    fd = df.detach()
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/reduction.py", line 189, in recv_handle
    return recvfds(s, 1)[0]
  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/reduction.py", line 159, in recvfds
    raise EOFError
EOFError

数据加载相关代码 #

from PIL import Image
from torch.utils.data import DataLoader

# extract of code of dataset
class Dataset():
    def __init__(self,image_files,mask_files):
        self.image_files = image_files
        self.mask_files = mask_files

    def __getitem__(self, idx):
        img = Image.open(self.image_files[idx]).convert('RGB')
        mask=Image.open(self.mask_files[idx]).convert('L')
        return img, mask

# extract of code of trainloader
train_loader = DataLoader(
                dataset=train_dataset,
                batch_size=4,
                num_workers=4,
                pin_memory=False,
                shuffle=True,
                drop_last=True,
                persistent_workers=False,
            )

恳请各位帮忙分析该错误的根源，谢谢！

内容的提问来源于stack exchange，提问作者rabbit-of-caerbannog