CUDA11.0部署指定PyTorch版本项目遇空张量min运算报错咨询

配置背景 #

• 目标开源项目的官方依赖要求：需搭配pytorch=0.4、python=3.7版本运行
• 本地硬件环境：显卡驱动搭载CUDA 11.0，最初安装的PyTorch版本与CUDA存在兼容性冲突
• 操作过程：完成适配版本的PyTorch安装后，运行项目触发异常报错

当前Conda环境包列表 #

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_pytorch_select           0.1                       cpu_0  
blas                      1.0                         mkl  
ca-certificates           2021.1.19            h06a4308_1  
certifi                   2020.12.5        py37h06a4308_0  
cffi                      1.14.0           py37h2e261b9_0  
cuda100                   1.0                           0    pytorch
cudatoolkit               9.2                           0  
freetype                  2.10.4               h5ab3b9f_0  
intel-openmp              2019.4                      243  
jpeg                      9b                   h024ee3a_2  
lcms2                     2.11                 h396b838_0  
libedit                   3.1.20210216         h27cfd23_1  
libffi                    3.2.1             hf484d3e_1007  
libgcc-ng                 9.1.0                hdf63c60_0  
libmklml                  2019.0.5                      0  
libpng                    1.6.37               hbc83047_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
libtiff                   4.2.0                h3942068_0  
libwebp-base              1.2.0                h27cfd23_0  
lz4-c                     1.9.3                h2531618_0  
mkl                       2020.2                      256  
mkl-service               2.3.0            py37he8ac12f_0  
mkl_fft                   1.3.0            py37h54f3939_0  
mkl_random                1.1.1            py37h0573a6f_0  
ncurses                   6.2                  he6710b0_1  
ninja                     1.10.2           py37hff7bd54_0  
numpy                     1.20.1                   pypi_0    pypi
numpy-base                1.19.2           py37hfa32c7d_0  
olefile                   0.46                     py37_0  
openssl                   1.1.1j               h27cfd23_0  
pillow                    8.1.2            py37he98fc37_0  
pip                       21.0.1           py37h06a4308_0  
pycparser                 2.20                       py_2  
python                    3.7.1                h0371630_7  
pytorch                   1.2.0           py3.7_cuda9.2.148_cudnn7.6.2_0    pytorch
readline                  7.0                  h7b6447c_5  
scipy                     1.6.1                    pypi_0    pypi
setuptools                52.0.0           py37h06a4308_0  
six                       1.15.0           py37h06a4308_0  
sqlite                    3.33.0               h62c20be_0  
tk                        8.6.10               hbc83047_0  
torchaudio                0.8.0                    pypi_0    pypi
torchvision               0.4.0                 py37_cu92    pytorch
typing                    3.7.4.3          py37h06a4308_0  
typing-extensions         3.7.4.3              hd3eb1b0_0  
typing_extensions         3.7.4.3            pyh06a4308_0  
wheel                     0.36.2             pyhd3eb1b0_0  
xz                        5.2.5                h7b6447c_0  
zlib                      1.2.11               h7b6447c_3  
zstd                      1.4.5                h9ceee32_0  

运行时报错栈 #

Model size: 44.17957M
==> Epoch 1/800  lr:0.0003
Traceback (most recent call last):
  File "/media/khawar/HDD_Khawar1/hypergraph_reid/main_video_person_reid_hypergraphsage_part.py", line 369, in <module>
    main()
  File "/media/khawar/HDD_Khawar1/hypergraph_reid/main_video_person_reid_hypergraphsage_part.py", line 230, in main
    train(model, criterion_xent, criterion_htri, optimizer, trainloader, use_gpu)
  File "/media/khawar/HDD_Khawar1/hypergraph_reid/main_video_person_reid_hypergraphsage_part.py", line 279, in train
    htri_loss = criterion_htri(features, pids)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/khawar/HDD_Khawar1/hypergraph_reid/losses.py", line 86, in forward
    dist_an.append(dist[i][mask[i] == 0].min())
RuntimeError: invalid argument 1: cannot perform reduction function min on tensor with no elements because the operation does not have an identity at /opt/conda/conda-bld/pytorch_1565287025495/work/aten/src/THC/generic/THCTensorMathReduce.cu:64

问题定位与解决方向 #

• 报错核心逻辑：计算困难三元组损失时，对当前样本筛选与自身身份不同的负样本后，得到空张量，直接在空张量上执行min()归约操作，触发CUDA计算层面的运行时错误
• 常见触发诱因：
  • 批次采样配置不合理：单个batch内样本身份过于单一，筛选后无有效负样本可选
  • 数据集标签映射错误：样本身份标记逻辑异常，导致负样本筛选条件全为True，无符合要求的候选样本
  • PyTorch版本行为差异：项目要求的pytorch=0.4与当前环境安装的pytorch=1.2.0对空张量的容错逻辑不一致，旧版本中无有效负样本时的处理分支在版本升级后失效
• 对应解决操作：
  • 调整DataLoader采样策略，确保每个训练batch内至少包含2个以上不同身份的样本，可采用身份均衡采样逻辑，每个batch固定抽取N个身份、每个身份固定数量的图像
  • 为losses.py第86行的负样本筛选逻辑增加空张量判断，若筛选后候选负样本长度为0，直接跳过当前样本的损失计算，避免空张量执行归约操作
  • 校验张量掩码计算逻辑，确认身份匹配的判断条件与距离矩阵维度完全对齐，避免维度不匹配导致的全掩码情况

内容的提问来源于stack exchange，提问作者Khawar Islam