训练时出现NaN/Inf张量警告及AddBackward0 RuntimeError求助

API 探路者

2026-8-22

问题

基于原运行正常的模型修改后，训练时出现以下异常：

未打乱输入数据的情况下，错误随机出现在数百到数千次迭代环节；
输入数据可在其他模型正常使用，但当前模型弹出NaN or Inf found in input tensor.警告，且在一个epoch结束前所有损失突然变为NaN；
怀疑模型层存在问题，但无法定位修复。

附模型forward代码：

def forward(self, x):
    # return self.model(x)
    x1_1 = self.cov1_1(x)
    x1_2 = self.cov1_2(x1_1)
    x1_3 = self.cov1_3(x1_2)
    x1_out = x1_1 + x1_3

    x2_1 = self.cov2_1(x1_out)
    x2_2 = self.cov2_2(x2_1)
    x2_3 = self.cov2_3(x2_2)
    x2_out = x2_1 + x2_3

    x3_1 = self.cov3_1(x2_out)
    x3_2 = self.cov3_2(x3_1)
    x3_3 = self.cov3_3(x3_2)
    x3_out = x3_1 + x3_3

    x4_1 = self.cov4_1(x3_out)
    x4_2 = self.cov4_2(x4_1)
    x4_3 = self.cov4_3(x4_2)
    x4_out = x4_1 + x4_3

    return {
        'x1_out': x1_out,
        'x2_out': x2_out,
        'x3_out': x3_out,
        'x4_out': x4_out,
    }

完整报错信息：

5%|▌         | 1498/27513 [1:12:34&lt;20:48:35,  2.88s/it]:
  5%|▌         | 1499/27513 [1:12:37&lt;20:39:44,  2.86s/it]:
  5%|▌         | 1500/27513 [1:12:40&lt;20:42:22,  2.87s/it]:
[W ..\\torch\\csrc\\autograd\\python_anomaly_mode.cpp:104] Warning: Error detected in AddBackward0. Traceback of forward call that caused the error:
  File \"D:\\codes\\220124_brainTumor\\tmi_220521\\MUNIT_tmi_seg _unifiedCC_lms220801\\train.py\", line 236, in &lt;module&gt;
    dices = trainer.gen_update(images, labels, config, missModality)
  File \"D:\\codes\\220124_brainTumor\\tmi_220521\\MUNIT_tmi_seg _unifiedCC_lms220801\\trainer.py\", line 357, in gen_update
    c, s_flair_prime, s_t1_prime, s_t1ce_prime, s_t2_prime = self.gen_uni.encode(x_miss_1_modality, missModality=missModality)
  File \"D:\\codes\\220124_brainTumor\\tmi_220521\\MUNIT_tmi_seg _unifiedCC_lms220801\\networks.py\", line 246, in encode
    contentsDic = self.enc_content(images)
  File \"D:\\programs\\Anaconda3\\envs\\MLAI\\lib\\site-packages\\torch\\nn\\modules\\module.py\", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File \"D:\\codes\\220124_brainTumor\\tmi_220521\\MUNIT_tmi_seg _unifiedCC_lms220801\\networks.py\", line 551, in forward
    x3_1 = self.cov3_1(x2_out)
  File \"D:\\programs\\Anaconda3\\envs\\MLAI\\lib\\site-packages\\torch\\nn\\modules\\module.py\", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File \"D:\\codes\\220124_brainTumor\\tmi_220521\\MUNIT_tmi_seg _unifiedCC_lms220801\\networks.py\", line 894, in forward
    x = self.conv(self.pad(x))
  File \"D:\\programs\\Anaconda3\\envs\\MLAI\\lib\\site-packages\\torch\\nn\\modules\\module.py\", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File \"D:\\programs\\Anaconda3\\envs\\MLAI\\lib\\site-packages\\torch\\nn\\modules\\conv.py\", line 446, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File \"D:\\programs\\Anaconda3\\envs\\MLAI\\lib\\site-packages\\torch\\nn\\modules\\conv.py\", line 442, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
 (function _print_stack)
  5%|▌         | 1500/27513 [1:12:43&lt;21:01:16,  2.91s/it]
Traceback (most recent call last):
  File \"D:\\codes\\220124_brainTumor\\tmi_220521\\MUNIT_tmi_seg _unifiedCC_lms220801\\train.py\", line 236, in &lt;module&gt;
    dices = trainer.gen_update(images, labels, config, missModality)
  File \"D:\\codes\\220124_brainTumor\\tmi_220521\\MUNIT_tmi_seg _unifiedCC_lms220801\\trainer.py\", line 488, in gen_update
    self.loss_gen_total.backward()
  File \"D:\\programs\\Anaconda3\\envs\\MLAI\\lib\\site-packages\\torch\\_tensor.py\", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File \"D:\\programs\\Anaconda3\\envs\\MLAI\\lib\\site-packages\\torch\\autograd\\__init__.py\", line 154, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Function 'AddBackward0' returned nan values in its 1th output.

Process finished with exit code 1

定位与修复方案

一、精准定位NaN产生位置

开启PyTorch异常检测，在训练代码开头添加：
```
torch.autograd.set_detect_anomaly(True)
```
该设置会在正向传播时就追踪NaN的产生，而非仅在反向传播报错时提示。

在forward函数的关键运算后添加断言检查，快速定位故障层：

def forward(self, x):
    x1_1 = self.cov1_1(x)
    assert not torch.isnan(x1_1).any(), "cov1_1输出含NaN"
    x1_2 = self.cov1_2(x1_1)
    assert not torch.isnan(x1_2).any(), "cov1_2输出含NaN"
    x1_3 = self.cov1_3(x1_2)
    assert not torch.isnan(x1_3).any(), "cov1_3输出含NaN"
    x1_out = x1_1 + x1_3
    assert not torch.isnan(x1_out).any(), "x1_out残差相加后含NaN"
    # 后续层同理添加断言检查
    ...

检查cov*模块的内部实现（对应报错第894行，包含pad和conv），确认激活函数、归一化层等是否存在数值异常的配置。

二、针对性修复方案

残差连接尺度不匹配：若残差分支（如x1_3）与主分支（x1_1）数值尺度差异过大，相加后易溢出。先打印两者的均值和方差：
```
print(f"x1_1: mean={x1_1.mean()}, std={x1_1.std()}")
print(f"x1_3: mean={x1_3.mean()}, std={x1_3.std()}")
```
若差异明显，可在残差分支添加1x1卷积调整尺度，或增加BN/LN层稳定数值。

权重初始化不当：新修改的卷积层若初始权重过大，会导致输出数值爆炸。改用保守的初始化方式：

# 在cov模块的__init__方法中添加
nn.init.kaiming_normal_(self.conv.weight, mode='fan_in', nonlinearity='relu')
nn.init.zeros_(self.conv.bias)

梯度爆炸：添加梯度裁剪，在反向传播前执行：

torch.nn.utils.clip_grad_norm_(self.gen_uni.parameters(), max_norm=1.0)

输入预处理问题：确认当前模型的输入预处理是否与原模型一致，比如是否将输入归一化到0-1或-1到1区间，避免卷积后数值过大。
小batch下的BN问题：若cov*模块包含BN层且batch size过小（如=1），会导致BN方差计算为0引发NaN。可替换为LayerNorm，或增大batch size。

内容的提问来源于stack exchange，提问作者squanchy

训练时出现NaN/Inf张量警告及AddBackward0 RuntimeError求助

问题 #

定位与修复方案 #

一、精准定位NaN产生位置 #

二、针对性修复方案 #

问题

定位与修复方案

一、精准定位NaN产生位置

二、针对性修复方案