当num_workers>0时最简DataLoader运行失败求助
DataLoader多Worker运行报错问题
以下极简DataLoader示例在num_workers=0时可正常运行,但将其设为1时会触发意外RuntimeError:RuntimeError: DataLoader worker (pid(s) 8248) exited unexpectedly。已将大型项目中的问题简化为该最小示例:
import torch from torch.utils.data import Dataset from torch.utils.data import DataLoader class DummyDataset(Dataset): def __init__(self, num_samples): self.num_samples = num_samples def __len__(self): return self.num_samples def __getitem__(self, idx): try: # Generate random data for features and labels # Features: shape (6, 64, 64), random float values # Labels: shape (64, 64), random binary values features = torch.randn(6, 64, 64) labels = torch.randint(0, 2, (64, 64)) return features, labels except Exception as e: print(f'Error at index {idx}: {e}') raise # Usage: num_samples = 1000 dummy_dataset = DummyDataset(num_samples) # Create a DataLoader dummy_dataloader = DataLoader(dummy_dataset, batch_size=32, shuffle=True, num_workers=1) # Try to fetch a batch of data data_batch, labels_batch = next(iter(dummy_dataloader)) print(data_batch.shape, labels_batch.shape)
本地环境基础配置正常,如下CIFAR10示例使用4个worker也无报错:
import torch import torchvision import torchvision.transforms as transforms transform = transforms.Compose([transforms.ToTensor()]) trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform) trainloader = torch.utils.data.DataLoader(trainset, batch_size=32, shuffle=True, num_workers=4) # Try to get a batch of data data, labels = next(iter(trainloader)) print(data.shape, labels.shape)
该示例在Google Colab中无问题,推测问题可能出在实现或本地未识别的环境问题上。环境版本:Python 3.11.5,torch 2.0.1,可提供更多信息用于排查。
内容的提问来源于stack exchange,提问作者Sonic
相关产品推荐
相关产品推荐

