使用双GPU的DataParallel训练速度远慢于单GPU的问题咨询

近期我在学习PyTorch多GPU模型训练，首先尝试了官方不推荐使用的DataParallel方法，构建了虚拟数据与简易模型，代码如下：

from datasets import load_dataset
from torch.utils.data import Dataset, DataLoader
import torch
import time

class My_dataset(Dataset):
    def __init__(self):
        self.x = torch.randn((15000, 768))
        self.y = torch.ones((15000,))
    
    def __getitem__(self, idx):
        return self.x[idx], self.y[idx]
    
    def __len__(self):
        return len(self.x)

class My_Model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = torch.nn.Linear(768, 76800)
        self.relu = torch.nn.ReLU()
        self.classfier = torch.nn.Linear(76800, 1)
        self.loss_fn = torch.nn.MSELoss()
    
    def forward(self, x):
        result = self.classfier(self.relu(self.linear1(x)))
        return result


train_dataset = My_dataset()
train_dataloader = DataLoader(train_dataset, batch_size=8, shuffle=True, drop_last=False)    


model = My_Model()
model = model.cuda()
# model = torch.nn.parallel.DataParallel(model, device_ids=[0,1])

loss_fn = torch.nn.MSELoss()

optimizer = torch.optim.Adam(params=model.parameters())
start = time.time()
for i, batch_datasets in enumerate(train_dataloader):
    if i % 100 == 0:
        print(f"i: {i}")
    x, y = batch_datasets
    x = x.cuda()
    y = y.cuda()
    
    result = model(x)
    loss = loss_fn(result, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
end = time.time()
print(f"time is {end - start}")    

测试结果：

• 单GPU训练耗时：2.7407000064849854秒，内存占用：1513MiB / 24268MiB
• 取消注释model = torch.nn.parallel.DataParallel(model, device_ids=[0,1])启用双GPU训练后，耗时：50.398988246917725秒，速度远慢于单GPU；双GPU内存占用为*[1785MiB / 24268MiB, 1331MiB / 24268MiB]*（batch size与单GPU保持一致）

我的疑问：

1. 为什么双GPU训练速度反而比单GPU慢这么多？
2. 上述双GPU的内存占用情况是否正常？

内容的提问来源于stack exchange，提问作者ShiZhou Huang