PyTorch自定义BCE损失出现NaN/-inf等异常值问题求助

已有Numpy实现的逻辑回归基础，现用PyTorch基于含2个特征的数据集，通过自动求梯度实现逻辑回归，代码如下：

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import torch
from sklearn.datasets import make_blobs

# making dataset
dataset = make_blobs(n_samples=200)
X = dataset[0]
y = dataset[1]

# for plotting the points
# plt.plot(X[y==0][:,0], X[y==0][:,1], '.')
# plt.plot(X[y==1][:,0], X[y==1][:,1], '.')

# making data the right shape
# X -> (n, m)
# y -> (1, m)
X = dataset[0].T
y = dataset[1].reshape(1, -1)

# converting to tensors 
X = torch.from_numpy(X)
y = torch.from_numpy(y)
X = X.type(torch.float32)
y = y.type(torch.float32)

# initializing weights, bias
n = X.shape[0]
m = X.shape[1]
W = torch.randn(n, 1, requires_grad = True)
b = torch.randn(1, requires_grad= True)

def sigmoid(x):
    return 1/(1 + torch.exp(-x))

def cost(X, y, W, b, m):
    fn = sigmoid( torch.matmul(W.T, X) + b )
    cost1 = y*torch.log(fn)
    cost2 = (1-y)*torch.log(1-fn)
    return (-1/m) * torch.sum(cost1 + cost2)

def logistic_regression(X, y, W, b, epochs=1000, learning_rate=0.0001):
    n = X.shape[0]
    m = X.shape[1]
    Y = y
    losses = []
    for i in range(epochs):
        loss = cost(X, y, W, b, m);
        losses.append(loss.item())
        
        loss.backward()
        
        with torch.no_grad():
            W -= learning_rate * W.grad
            b -= learning_rate * b.grad
            
        W.grad.zero_()
        b.grad.zero_()
        
    return losses

losses = logistic_regression(X, y, W, b)

决策边界绘制代码：

# To plot the decision boundary

theta = W.detach().numpy()
bias = b.detach().numpy()

X = dataset[0]
y = dataset[1]

# Do change the value of test_points according to the dataset
test_points = np.linspace(-10, 10, 100)
# the equation is -> theta0 * x1 + theta1 * x2 + b = 0
# we are plotting x1 (x-axis) vs x2 (y-axis)
# so x2 = (-theta0 * x1 - b)/theta1

y_points = (-theta[0]*test_points - bias)/theta[1]
plt.plot(X[y==0][:,0], X[y==0][:,1], '.')
plt.plot(X[y==1][:,0], X[y==1][:,1], '.')
plt.plot(test_points, y_points)

运行时发现自定义BCE损失有时会出现NaN、-inf或负损失值，请问错误原因是什么？

• 对数函数定义域越界：当模型的sigmoid输出fn趋近于0或1时，torch.log(fn)或torch.log(1-fn)会计算出-inf（log(0)的极限为负无穷），后续求和再乘以-1/m时，会直接引发NaN或-inf的数值异常；若计算过程中极端值未完全传播，还可能出现不符合预期的负损失。

• 数值稳定性缺失：手动实现的sigmoid函数在输入值绝对值过大时会出现溢出：

  • 输入为极大正值时，torch.exp(-x)趋近于0，sigmoid输出1；
  • 输入为极大负值时，torch.exp(-x)爆炸式增长，sigmoid输出0。
    两种情况都会触发后续对数计算的异常。

• 学习率设置不合理：当前使用的learning_rate=0.0001过小，模型收敛速度极慢，参数长时间处于随机初始化的不稳定状态，容易在迭代中出现极端预测值；若学习率过大则可能引发梯度爆炸，直接导致参数变为NaN。

1. 改用PyTorch内置损失函数：推荐使用torch.nn.BCEWithLogitsLoss，它将sigmoid激活和BCE损失合并计算，通过log-sum-exp技巧避免数值不稳定，无需手动实现sigmoid和对数计算：

   # 替换自定义cost函数和训练逻辑
   loss_fn = torch.nn.BCEWithLogitsLoss()
   
   def logistic_regression(X, y, W, b, epochs=1000, learning_rate=0.01):
       losses = []
       for i in range(epochs):
           logits = torch.matmul(W.T, X) + b
           loss = loss_fn(logits, y)
           losses.append(loss.item())
           
           loss.backward()
           
           with torch.no_grad():
               W -= learning_rate * W.grad
               b -= learning_rate * b.grad
               
           W.grad.zero_()
           b.grad.zero_()
           
       return losses
   

2. 手动实现时添加数值截断：如果坚持手动实现损失，需要对sigmoid输出做截断，避免进入对数函数的无效定义域：

   def cost(X, y, W, b, m):
       fn = sigmoid( torch.matmul(W.T, X) + b )
       # 截断fn到[1e-8, 1-1e-8]区间，避免log(0)或log(1)
       fn = torch.clamp(fn, 1e-8, 1 - 1e-8)
       cost1 = y*torch.log(fn)
       cost2 = (1-y)*torch.log(1-fn)
       return (-1/m) * torch.sum(cost1 + cost2)
   

3. 调整学习率：将学习率调整为合理范围（如0.01），加快模型收敛，减少参数处于不稳定状态的时间；也可配合梯度裁剪进一步防止梯度爆炸。

内容的提问来源于stack exchange，提问作者varun