如何提升三分类任务下前馈神经网络（Feed Forward NN）的准确率？

Hey there! Congrats on building your first neural network—that’s such a great first step, even if the performance isn’t where you want it right now. That ~0.37 accuracy is barely better than random guessing (which would be ~0.33 for 3 classes), so let’s walk through the most likely issues and fixes to get your model learning properly.

1. Fix Data Preprocessing First #

Neural networks are extremely sensitive to feature scaling—this is probably the most common culprit for stuck performance with simple models.

• Normalize/Standardize Your Features: Your X is taken directly from the dataset without any scaling. If your features have wildly different ranges (e.g., one feature is 0-1000 and another is 0-1), the gradient updates will be unstable, and the model can’t learn meaningful patterns. Try scaling X to a consistent range:

  # Option 1: Standardize to mean 0, std 1
  X = (X - np.mean(X, axis=0)) / np.std(X, axis=0)
  # Option 2: Normalize to 0-1 range
  X = (X - np.min(X, axis=0)) / (np.max(X, axis=0) - np.min(X, axis=0))
  

• Verify One-Hot Encoding: Double-check that your Y matrix is correctly mapped to your labels. If your labels are 0/1/2, make sure each row has exactly one 1 in the corresponding column. Here’s a safe way to implement it:

  labels = labels.astype(int)  # Ensure labels are integers
  Y = np.zeros((m, 3))
  for i in range(m):
      Y[i, labels[i]] = 1
  

  A mistake here (e.g., misaligning labels to columns) would make the model learn totally wrong patterns.

2. Use the Right Loss Function & Output Activation #

For 3-class classification with one-hot labels:

• Output Layer: Must use a softmax activation—this ensures the output sums to 1, representing class probabilities. If you’re using sigmoid (for binary classification) or no activation, your model can’t properly model multi-class probabilities.
• Loss Function: Use categorical cross-entropy loss instead of mean squared error (MSE). MSE is designed for regression tasks and is inefficient for classification, especially with softmax outputs. For a numpy implementation, here’s a stable version (to avoid log(0) errors):

  def categorical_crossentropy(y_pred, y_true):
      epsilon = 1e-10
      y_pred = np.clip(y_pred, epsilon, 1 - epsilon)
      return -np.sum(y_true * np.log(y_pred)) / y_pred.shape[0]
  

3. Validate Gradient Calculations (Critical for Manual Implementations) #

If you’re coding the network from scratch (not using a framework like TensorFlow/PyTorch), it’s extremely easy to mess up gradient calculations. A wrong gradient means adjusting parameters in the wrong direction—so no amount of tuning learning rate or hidden units will help.

• Check Gradients Numerically: For each weight/bias parameter, compute the numerical gradient (by slightly perturbing the parameter and measuring loss change) and compare it to your analytical gradient. They should be almost identical (error < 1e-7). Example for a weight matrix W1:

  def compute_numerical_gradient(W, X, Y, loss_fn, predict_fn):
      eps = 1e-5
      grad_num = np.zeros_like(W)
      for i in range(W.shape[0]):
          for j in range(W.shape[1]):
              W[i,j] += eps
              loss_plus = loss_fn(predict_fn(X, W), Y)
              W[i,j] -= 2*eps
              loss_minus = loss_fn(predict_fn(X, W), Y)
              grad_num[i,j] = (loss_plus - loss_minus) / (2*eps)
              W[i,j] += eps
      return grad_num
  

  Compare this to your analytical gradient—if they don’t match, fix your backprop code first.

4. Tune Training Hyperparameters (Wisely) #

You tried adjusting learning rate and hidden units, but let’s make sure you’re testing the right ranges:

• Learning Rate: Try values spanning several orders of magnitude: 1e-4, 1e-3, 1e-2, 1e-1. Plot your training loss over epochs—if loss bounces around wildly, your learning rate is too big; if loss stays flat for hundreds of epochs, it’s too small.
• Training Epochs: Are you training long enough? If you’re only doing 50-100 epochs, the model might not have had time to converge. Keep training until the loss stops decreasing (plateaus) on a validation set.
• Hidden Layer Activation: If you’re using a linear activation in the hidden layer, your entire model is just a linear classifier—completely unable to learn non-linear patterns. Switch to ReLU (most common) or tanh for non-linearity.

5. Fix Parameter Initialization #

Bad initialization can cause gradient vanishing/exploding right out the gate:

• Avoid initializing weights to 0 (this makes all neurons learn the same thing) or large random values. Use:
  • Xavier Initialization (for tanh/sigmoid): W = np.random.randn(n_in, n_out) * np.sqrt(1/n_in)
  • He Initialization (for ReLU): W = np.random.randn(n_in, n_out) * np.sqrt(2/n_in)

Start with these fixes—odds are one of these (especially data scaling, gradient checks, or loss function choice) is the root cause. Once you get the training loss decreasing, you can start experimenting with more hidden units or regularization later.

内容的提问来源于stack exchange，提问作者Highness