为何GPU上torchvision模型Eval模式下不同batch size输出不同？

问题现象 #

• 测试发现，使用torchvision的EfficientNetV2-S、VGG11等模型时，即使处于eval()模式，不同batch size的输入会产生细微的输出差异，且EfficientNetV2-S的差异更显著。
• CPU环境下输出完全一致，仅GPU环境下出现该问题。

测试代码与结果 #

初始测试代码 #

import torch
import torchvision

model = torchvision.models.vgg11(weights = None).cuda()
model.eval()
model = model.features

inputs = torch.rand([128, 3, 128, 128]).cuda()

# small batch size
out1 = model(inputs[:2])

# large batch size
out2 = model(inputs)[:2]

CPU补充测试（结果一致） #

torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

device = 'cpu'

model = torchvision.models.vgg11(weights = None).to(device = device)
model.eval()
model = model.features

inputs = torch.rand([128, 3, 128, 128]).to(device = device)

# small batch size
out1 = model(inputs[:2])
# large batch size
out2 = model(inputs)[:2]
print(torch.all(torch.eq(out1, out2)))

model = torch.nn.Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)).to(device = device)
model.eval()
out1 = model(inputs[:2])
out2 = model(inputs)[:2]
print(torch.all(torch.eq(out1, out2)))

model = torch.nn.ReLU(inplace=True).to(device = device)
model.eval()
out1 = model(inputs[:2])
out2 = model(inputs)[:2]
print(torch.all(torch.eq(out1, out2)))

model = torch.nn.MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False).to(device = device)
model.eval()
out1 = model(inputs[:2])
out2 = model(inputs)[:2]
print(torch.all(torch.eq(out1, out2)))

# 输出结果
# tensor(True)
# tensor(True)
# tensor(True)
# tensor(True)

GPU补充测试（模型整体输出不一致） #

torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

device = 'cuda'

model = torchvision.models.vgg11(weights = None).to(device = device)
model.eval()
model = model.features

inputs = torch.rand([128, 3, 128, 128]).to(device = device)

# small batch size
out1 = model(inputs[:2])
# large batch size
out2 = model(inputs)[:2]
print(torch.all(torch.eq(out1, out2)))

model = torch.nn.Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)).to(device = device)
model.eval()
out1 = model(inputs[:2])
out2 = model(inputs)[:2]
print(torch.all(torch.eq(out1, out2)))

model = torch.nn.ReLU(inplace=True).to(device = device)
model.eval()
out1 = model(inputs[:2])
out2 = model(inputs)[:2]
print(torch.all(torch.eq(out1, out2)))

model = torch.nn.MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False).to(device = device)
model.eval()
out1 = model(inputs[:2])
out2 = model(inputs)[:2]
print(torch.all(torch.eq(out1, out2)))

# 输出结果
# tensor(False, device='cuda:0')
# tensor(True, device='cuda:0')
# tensor(True, device='cuda:0')
# tensor(True, device='cuda:0')

原因分析 #

1. 多层运算的浮点误差累积：单独测试单个卷积、ReLU、池化层时结果一致，但VGG11的features模块是多层操作的组合。GPU处理不同batch size时，底层CUDA/cuDNN会选择不同的并行优化策略（比如大batch时的张量分组处理），这些策略在浮点运算的舍入精度上存在细微差异，经过多层传递后，误差被放大到可观测的程度。
2. GPU硬件的计算特性：GPU的SIMD架构在处理不同规模的张量时，会调度不同的计算单元组合，导致浮点运算的执行路径不同，进而产生不一致的舍入误差。这种误差在CPU上不会出现，因为CPU的浮点运算逻辑更统一，且没有大规模并行调度带来的路径差异。
3. cuDNN确定性设置的局限性：虽然开启了cudnn.deterministic = True，但该设置无法覆盖所有CUDA算子的优化路径。部分复杂的多层组合操作，在不同batch size下仍可能触发不同的底层实现，导致输出差异。
4. 模型复杂度的影响：EfficientNetV2-S包含更多深度可分离卷积、注意力机制等复杂操作，这些操作对浮点误差的敏感度更高，因此差异比VGG11更显著。

内容的提问来源于stack exchange，提问作者Jake