PVT模型无论输入何种数据均输出固定值的问题求助

我最近用IDMM算法（一种AI预训练算法）训练了一个PVT模型，使用的数据集包含3个类别，总共约2800张图片。原本训练500个epoch后，我预期模型准确率能达到80%-90%，结果实际只有34%。

深入排查后发现了核心问题：不管输入什么样的图像，模型输出的结果都是固定的tensor([[ 0.1031, 0.0980, -0.2104]]——哪怕单独输入单张图片测试，结果也完全一样。

下面是我用来复现问题的测试代码：

import torch
import torchvision
import VIT.models.pvt_modelv2 as KEVIN_pvtv2
from collections import OrderedDict

num1 = torch.rand(32,3,224,224)
num2 = torch.rand(32,3,224,224)
print(num1, num2)

checkpoint = torch.load("C:\\Users\\vvenkata\\Desktop\\coding\\KEVIN\\model\\VIT\\checkpoints\\224_finetune\\Training\\pvt_v2_b3_lr_0.05_wd_0.0013_bs_16_epochs_200_dim_192_cutmix_0.7_path_Training\\pretrained_pvt_v2_b3_lr_0.054_w\\model_best.pth.tar", map_location='cuda:0')
pvt = KEVIN_pvtv2.pvt_v2_b3(num_classes=3)    
new_state_dict = OrderedDict()
for k, v in checkpoint.items():
    name = k[7:] if k.startswith("module.") else k
    new_state_dict[name] = v
pvt.load_state_dict(new_state_dict)
pvt.eval()

output1 = pvt(num1)
output2 = pvt(num2)
print(f"Output1 {output1}, output2 {output2}")

测试时的输入和输出示例如下：

input:  [[[0.6883, 0.8841, 0.8812,  ..., 0.2528, 0.6816, 0.1573],
          [0.4715, 0.0426, 0.7539,  ..., 0.3806, 0.3291, 0.4134],
          [0.1406, 0.2837, 0.1914,  ..., 0.6967, 0.9686, 0.9358],
          ...,
          [0.9271, 0.6876, 0.2966,  ..., 0.8610, 0.3527, 0.9421],
          [0.8541, 0.4442, 0.5698,  ..., 0.4103, 0.9373, 0.9191],
          [0.5845, 0.6987, 0.1419,  ..., 0.3499, 0.8706, 0.8108]],

         [[0.7734, 0.1418, 0.2561,  ..., 0.9039, 0.5064, 0.8974],
          [0.9297, 0.5981, 0.6904,  ..., 0.5158, 0.0603, 0.4466],
          [0.7587, 0.5091, 0.2655,  ..., 0.7955, 0.5502, 0.9425],
          ...,
          [0.4653, 0.0370, 0.9665,  ..., 0.6748, 0.5855, 0.4120],
          [0.6278, 0.7346, 0.1757,  ..., 0.9457, 0.8254, 0.5958],
          [0.8474, 0.2402, 0.7168,  ..., 0.2539, 0.8296, 0.0168]]...

output:  [ 0.1031,  0.0980, -0.2104],
        [ 0.1031,  0.0980, -0.2104],
        [ 0.1031,  0.0980, -0.2104],
        [ 0.1031,  0.0980, -0.2104],
        [ 0.1031,  0.0980, -0.2104],
        [ 0.1031,  0.0980, -0.2104],
        [ 0.1031,  0.0980, -0.2104],
        [ 0.1031,  0.0980, -0.2104], ...

我可以确定不是数据加载的问题，因为这个最简测试示例也能复现问题。请问有没有大佬能指点下该怎么解决这个问题？

备注：内容来源于stack exchange，提问作者KEVIN