手写数字识别神经网络输出高估及梯度爆炸问题求助

Hey Eric, let's work through this problem together—gradient explosion and wonky overestimated outputs are super common when you're first building MNIST models, so we've got clear, actionable fixes to try:

1. Fix Your Output Layer Activation #

Your output layer is probably using a linear activation right now, which is why outputs are blowing up without bounds. Swap it for a softmax activation:

• Softmax converts raw outputs into a 0-1 probability distribution, perfectly matching your 0-9 classification task. It ensures the sum of all 10 output neurons equals 1, so you won't get those inflated, nonsensical values anymore.

2. Tune Weight Initialization #

Gradient explosion often starts with overly large initial weights. Ditch random 0-1 weights and use targeted initialization:

• For sigmoid/tanh hidden layers: Use Xavier initialization

  weights_input_hidden = np.random.randn(784, 15) * np.sqrt(1 / 784)
  

• For ReLU hidden layers (I recommend switching to ReLU if you haven't already—it's more stable): Use He initialization

  weights_input_hidden = np.random.randn(784, 15) * np.sqrt(2 / 784)
  

These methods scale weights based on the number of input neurons, preventing outputs from blowing up early in training.

3. Add Gradient Clipping #

This is the most direct fix for gradient explosion. After calculating gradients during backprop, cap their magnitude to prevent runaway updates:

max_gradient_norm = 1.0
# Calculate total gradient norm across all weights/biases
total_norm = np.sqrt(np.sum([np.sum(np.square(grad)) for grad in [w1_grad, w2_grad, b1_grad, b2_grad]]))
if total_norm > max_gradient_norm:
    scale_factor = max_gradient_norm / total_norm
    # Rescale all gradients to stay within the norm limit
    w1_grad *= scale_factor
    w2_grad *= scale_factor
    b1_grad *= scale_factor
    b2_grad *= scale_factor

This keeps gradients from spiraling out of control during weight updates.

4. Lower Your Learning Rate #

A learning rate that's too high makes weight updates way too aggressive, worsening both gradient explosion and output instability. Try dropping it from your current value (e.g., if you're using 0.1, go to 0.01 or even 0.001) and see if outputs stabilize.

5. Double-Check Forward Pass Logic #

That first wrong prediction might hint at a basic calculation error:

• Verify your weight matrix dimensions: Input (784) → Hidden (15) needs a (784, 15) weight matrix, not (15,784) (easy mix-up!)
• Test with a simple input (like all 0s or all 1s) and manually compute the forward pass to match against your code's output. This will catch any matrix multiplication or activation function bugs.

6. Add L2 Regularization (Optional) #

L2 regularization penalizes large weights, which helps prevent gradient drift long-term:

# Add this term to your loss calculation
l2_lambda = 0.001
loss = cross_entropy_loss + l2_lambda * (np.sum(np.square(w1)) + np.sum(np.square(w2)))

Tweak l2_lambda (start small!) to find a balance between preventing overfitting and keeping weights stable.

内容的提问来源于stack exchange，提问作者Eric Dampierre