Python反向传播代码疑问：delta3[range(num_examples), y] -= 1含义解析

Let’s break down this line clearly—it’s a key, memory-efficient step in computing gradients for cross-entropy loss paired with a softmax output layer.

First, let’s recap the context of your code:

• probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True) calculates the softmax of the model’s raw logits, converting them into a probability distribution where each row (per sample) sums to 1.
• delta3 = probs initializes delta3 to store the gradient of the loss function with respect to the softmax outputs.

Now, the line delta3[range(num_examples), y] -= 1:

• range(num_examples) generates an index for every sample in your batch (from 0 to num_examples-1).
• y is the array of true class labels for each sample (e.g., [0, 2, 1] for a 3-class problem with 3 samples).
• Together, these indices target the exact position in delta3 that corresponds to the true class for each sample. Subtracting 1 from those positions transforms delta3 into probs - y_true, where y_true is the one-hot encoded version of your labels.

Why this works #

For cross-entropy loss (the standard loss for classification tasks), the gradient of the loss with respect to the softmax outputs is exactly probs - y_true. Instead of explicitly creating a one-hot matrix (which can waste memory for large batches or many classes), this line modifies the softmax output array directly:

• For each sample, all non-true class positions stay as their original softmax probability.
• The true class position becomes probs[sample_idx, true_class] - 1.

This resulting delta3 is then used to compute gradients for the previous layer’s weights (like dW2 = (a1.T).dot(delta3)), which is the core of backpropagation.

内容的提问来源于stack exchange，提问作者lelli