You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

当num_workers>0时最简DataLoader运行失败求助

DataLoader多Worker运行报错问题

以下极简DataLoader示例在num_workers=0时可正常运行,但将其设为1时会触发意外RuntimeError:RuntimeError: DataLoader worker (pid(s) 8248) exited unexpectedly。已将大型项目中的问题简化为该最小示例:

import torch
from torch.utils.data import Dataset
from torch.utils.data import DataLoader

class DummyDataset(Dataset):
    def __init__(self, num_samples):
        self.num_samples = num_samples

    def __len__(self):
        return self.num_samples

    def __getitem__(self, idx):
        try:
            # Generate random data for features and labels
            # Features: shape (6, 64, 64), random float values
            # Labels: shape (64, 64), random binary values
            features = torch.randn(6, 64, 64)
            labels = torch.randint(0, 2, (64, 64))
            return features, labels
        except Exception as e:
            print(f'Error at index {idx}: {e}')
            raise

# Usage:
num_samples = 1000
dummy_dataset = DummyDataset(num_samples)

# Create a DataLoader
dummy_dataloader = DataLoader(dummy_dataset, batch_size=32, shuffle=True, num_workers=1)

# Try to fetch a batch of data
data_batch, labels_batch = next(iter(dummy_dataloader))
print(data_batch.shape, labels_batch.shape)

本地环境基础配置正常,如下CIFAR10示例使用4个worker也无报错:

import torch
import torchvision
import torchvision.transforms as transforms

transform = transforms.Compose([transforms.ToTensor()])
trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=32, shuffle=True, num_workers=4)

# Try to get a batch of data
data, labels = next(iter(trainloader))
print(data.shape, labels.shape)

该示例在Google Colab中无问题,推测问题可能出在实现或本地未识别的环境问题上。环境版本:Python 3.11.5,torch 2.0.1,可提供更多信息用于排查。

内容的提问来源于stack exchange,提问作者Sonic

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.07.09 22:22:11