PyTorch nn.DataParallel报错：输入与权重张量类型不匹配

问题现象 #

在单机多GPU环境中使用torch.nn.DataParallel时，出现设备不匹配报错：

RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

问题根源 #

代码存在两个核心错误：

1. 未将原始模型model_ft迁移到GPU，直接用CPU模型创建DataParallel，且未将包装后的模型赋值给model_ft，导致训练时模型权重仍在CPU，输入数据却已迁移到GPU，二者设备不一致。
2. 优化器基于CPU版本的model_ft创建，无法正确更新多GPU模型的参数。

修复步骤 #

1. 调整模型迁移与DataParallel包装顺序 #

先将模型迁移到GPU，再用DataParallel包装，并将包装后的模型赋值给model_ft，确保后续所有操作都基于多GPU模型：

修改前代码：

# Utilize multiple GPUS
if 'cuda' in device:
    print(device)
    print("using data parallel")
    net = torch.nn.DataParallel(model_ft) # make parallel
    cudnn.benchmark = True

# Transfer the model to GPU
#model_ft = model_ft.to(device)

修改后代码：

# 先将模型迁移到GPU，再包装DataParallel
if 'cuda' in device:
    print(device)
    model_ft = model_ft.to(device)
    print("using data parallel")
    model_ft = torch.nn.DataParallel(model_ft)
    cudnn.benchmark = True

2. 确保损失函数与设备一致（可选但推荐） #

将损失函数也迁移到GPU，避免潜在的设备不匹配问题：

# Loss function
criterion = nn.CrossEntropyLoss()
if 'cuda' in device:
    criterion = criterion.to(device)

3. 验证优化器创建时机 #

优化器必须基于已经迁移到GPU并包装好的model_ft创建，确保能正确获取GPU上的模型参数。修改后的代码中，优化器创建在模型迁移和包装之后，无需额外调整，但需确保顺序正确。

完整修改后关键代码片段 #

# 模型迁移与多GPU包装
if 'cuda' in device:
    print(device)
    model_ft = model_ft.to(device)
    print("using data parallel")
    model_ft = torch.nn.DataParallel(model_ft)
    cudnn.benchmark = True

# Loss function
criterion = nn.CrossEntropyLoss()
if 'cuda' in device:
    criterion = criterion.to(device)

# Optimizer - 基于包装后的多GPU模型创建
optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

# Learning rate decay
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

报错回溯（中文翻译） #

Traceback (most recent call last):
  File "/home2/coremax/Documents/pytorch-image-classification/train.py", line 263, in <module>
    model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
  File "/home2/coremax/Documents/pytorch-image-classification/train.py", line 214, in train_model
    outputs = model(inputs)
  File "/home2/coremax/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home2/coremax/anaconda3/lib/python3.9/site-packages/timm/models/resnet.py", line 730, in forward
    x = self.forward_features(x)
  File "/home2/coremax/anaconda3/lib/python3.9/site-packages/timm/models/resnet.py", line 709, in forward_features
    x = self.conv1(x)
  File "/home2/coremax/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home2/coremax/anaconda3/lib/python3.9/site-packages/torch/nn/modules/container.py", line 217, in forward
    input = module(input)
  File "/home2/coremax/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home2/coremax/anaconda3/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 463, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home2/coremax/anaconda3/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: 输入类型(torch.cuda.FloatTensor)与权重类型(torch.FloatTensor)不匹配

内容的提问来源于stack exchange，提问作者Khawar Islam