从零搭建神经网络无法拟合小样本？求排查反向传播问题

我手动实现了仅用sigmoid激活函数的两层神经网络，尝试拟合小样本训练数据，但训练后无论输入什么，模型始终输出固定类分布；移除隐藏层改用单层神经网络时，效果却好很多。

我知道正确实现应全程使用向量化运算，但想先手动实现以深入理解训练的每一个环节，怀疑反向传播存在错误，反复检查却未能发现问题。

补充说明：我了解仅使用sigmoid可能存在梯度消失问题，但仅两层隐藏层应该仍可正常工作，希望能在仅用sigmoid和手动运算的前提下解决问题。

实现代码 #

import numpy as np
import pandas as pd
from matplotlib import pyplot as plt

data = pd.read_csv('train.csv')

# Training data management
data= np.array(data)

# Train test split 80:20
test_datas = data[int(len(data)*0.8):]
train_datas = data[:int(len(data)*0.8)]

# Separating pixel data and label data
train_labels = train_datas[:,0] # label col
train_datas = (train_datas[:,1:] - np.min(train_datas[:,1:]))/(np.max(train_datas[:,1:])-np.min(train_datas[:,1:])) # pixel data, scaled to 0-1

test_labels = test_datas[:,0] # label col
test_datas = (test_datas[:,1:] - np.min(test_datas[:,1:]))/(np.max(test_datas[:,1:])-np.min(test_datas[:,1:])) # pixel data, scaled to 0-1

def sigmoid(x): # sigmoid func to squish all inputs into range 0 to 1
    return 1 / (1 + np.exp(-x))

# Initialization
size=[16, 10]
train_data = train_datas[:10] 
train_label = train_labels

weights = [] # list to store all the weights for every layer
biases = [] # list to store all the biases for every layer

# Try using Xavier/Glorot initialization
for i in range(len(size)): # Initialize weights for each layer
    if i == 0:
        weights.append(np.random.randn(size[0], len(train_data[0])) * np.sqrt(1/len(train_data[0])))
    else:
        weights.append(np.random.randn(size[i], size[i-1]) * np.sqrt(1/size[i-1]))

for i in range(len(size)):  # Initialize biases for each layer
    if i == 0:
        biases.append(np.zeros(size[0])) # First layer biases
    else:
        biases.append(np.zeros(size[i]))  

# Temporarily training on 10 data example for trouble shooting
learning_rate = 0.1
for w in range(1):
    train_data = train_datas[w*10:(w+1)*10] 
    for o in range(10):
        cost = 0

        # Create temporary storage for averaging weights and biases
        temp_weights = [] 
        temp_biases = [] 

        temp_weights.append(np.zeros(shape=(size[0],len(train_data[0])))) # First layer
        temp_biases.append(np.zeros(size[0])) 
        for i in range(len(size)-1): 
            temp_weights.append(np.zeros(shape=(size[i+1],size[i]))) # following layers
            temp_biases.append(np.zeros(size[i+1])) 

        for i in range(len(train_data)): # Iterate through every train_data
            # Forward propagation
            Zs = []
            As = [train_data[i]] 
            z = weights[0] @ train_data[i] + biases[0] # First layer
            a = sigmoid(z)
            Zs.append(z) # Storing data for backward propagation
            As.append(a)
            
            for j in range(len(size)-1): 
                z = weights[j+1] @ a + biases[j+1] # Following layers
                a = sigmoid(z)
                Zs.append(z) # Storing data for backward propagation
                As.append(a)

            # Calculating cost
            one_hot = np.zeros(10)
            one_hot[train_label[i]]=1
            
            cost = cost + np.sum((a - one_hot)**2) # Just to keep track of model fit

            # final/output layer Backpropagation
            dC_da = 2*(a - one_hot) 
            dadz = (np.exp(-z) / (1 + np.exp(-z))**2)
            
            for x in range (len(weights[-1][0])): # iterating through weights column by column
                # updating weights              
                dzdw = As[-2][x] # This one input, affects a whole column of weights
                dC_dw = dC_da * dadz * dzdw 

                (temp_weights[-1])[:,x] += -dC_dw*learning_rate/len(train_data) # keeping track of updates to the weights
                
            # updating Biases
            dzdb = 1
            dC_db = dC_da * dadz * dzdb
            temp_biases[-1] += -dC_db*(learning_rate)/len(train_data) # keeping track of updates to the biases

            # Previous layer Backpropagation
            dCda_0 = np.array([])
            for x in range (len(weights[-1][0])): # iterating through inputs, a, summing weights column by column, 
                dzda_0 = weights[-1][:,x] # A whole column of weights affect how ONE prev layer input affects the next layer 
                dC_da_0 = np.sum(dC_da*dadz*dzda_0)/len(weights[-1]) # Keep track of how previous layer output affect next layer for chain rule later
                dCda_0 = np.append(dCda_0,dC_da_0)

            # Previous layer weights
            for k in range(len(size)-1): # iterating through layers, starting from the second last
                z = Zs[-k-2]
                dadz = (np.exp(-z) / (1 + np.exp(-z))**2)
                
                # Updating previous layer weights
                for l in range (len(weights[-2-k][0])): # iterating through weights column by column (-2-k because we start from second from last)
                    
                    dzdw = As[-3-k][l] # This one input, affects a whole column of weights
                    dC_dw = dCda_0 * dadz * dzdw
                    
                    (temp_weights[-2-k])[:,l] += -dC_dw*(learning_rate)/len(train_data) # keeping track of updates to the weights

                # updating Biases
                dzdb = 1
                dC_db = dCda_0 * dadz * dzdb
                temp_biases[-2-k] += -dC_db*(learning_rate)/len(train_data) # keeping track of updates to the biases

                # Keep track of how this layer output affect next layer for chain rule later
                temp_dCda_0 = np.array([])
                for x in range (len(weights[-2-k][0])): # iterating through inputs, a, summing weights column by column
                    dzda_0 = weights[-2-k][:,x] # A whole column of weights affect how ONE prev layer input affects the next layer 
                    dC_da_0 = np.sum(dCda_0*dadz*dzda_0)/len(weights[-2-k]) 
                    temp_dCda_0 = np.append(temp_dCda_0,dC_da_0)
                
                dCda_0 = temp_dCda_0 

        # Updating biases and weights
        for i in range(len(size)):
            weights[i] += temp_weights[i]
            biases[i] += temp_biases[i]

        # Analysis of changes to weights 
        print("weights, iteration",o)
        print(temp_weights[0][0][132:136])
        print("\n", weights[0][0][132:136])
        print("\n",temp_weights[1][0])
        print("\n", weights[1][0])

        # Analysis of changes to biases 
        print("biases, iteration",o)
        print("\n",temp_biases[0])
        print("\n", biases[0])
        print("\n", temp_biases[1])
        print("\n", biases[1])

# Forward propagation, testing training fit
m=0
z = weights[0] @ train_datas[m] + biases[0] # First layer
a = sigmoid(z)
print("\nFirst layer, \nz=",z,"\na=",a )

for j in range(len(size)-1): 
    z = weights[j+1] @ a + biases[j+1] # Following layers
    a = sigmoid(z)
    print("\n",j+1,"th layer, \nz=",z,"\na=",a )

print("\nevaluation=",a,"max= ",np.argmax(a)," label= ",train_labels[m])

# Forward propagation, testing training fit
m=4
z = weights[0] @ train_datas[m] + biases[0] # First layer
a = sigmoid(z)
print("\nFirst layer, \nz=",z,"\na=",a )

for j in range(len(size)-1): 
    z = weights[j+1] @ a + biases[j+1] # Following layers
    a = sigmoid(z)
    print("\n",j+1,"th layer, \nz=",z,"\na=",a )

print("\nevaluation=",a,"max= ",np.argmax(a)," label= ",train_labels[m])

# Check accuracy on training set
correct = 0
k = 100
for i in range(k):
    z = weights[0] @ train_datas[i] + biases[0] # First layer
    a = sigmoid(z)

    for j in range(len(size)-1): 
        z = weights[j+1] @ a + biases[j+1] # Following layers
        a = sigmoid(z)

    if train_labels[i] == np.argmax(a): 
        correct += 1
        
print(correct/k)

核心问题排查与修正 #

1. 梯度计算的错误缩放 #

反向传播中计算前一层神经元梯度时，你错误地对求和结果做了除法（/len(weights[-1])、/len(weights[-2-k])），这会导致梯度被不必要地缩小，直接影响权重更新的有效性：

# 错误代码
dC_da_0 = np.sum(dC_da*dadz*dzda_0)/len(weights[-1])
# 修正后
dC_da_0 = np.sum(dC_da*dadz*dzda_0)

反向传播中，前一层每个神经元的梯度是后一层所有相关梯度的加权和，不需要额外做平均操作。

2. 样本与标签不匹配 #

训练时train_label直接使用了整个训练集的标签，但train_data只取了10个样本，导致样本和标签对应错误：

# 错误代码
train_label = train_labels
# 修正后，取对应批次的标签
train_label = train_labels[w*10:(w+1)*10]

3. Sigmoid导数计算优化 #

原代码中sigmoid导数的计算可以简化为a * (1 - a)，既高效又避免数值不稳定：

# 替换原来的dadz计算
dadz = a * (1 - a)

4. 训练迭代次数不足 #

当前仅做了10次迭代，对于两层网络来说远远不够，建议将迭代次数提升至1000次以上，同时可以适当调高学习率（比如0.5或1.0），弥补手动计算的效率问题。

内容的提问来源于stack exchange，提问作者Turtleishly