请求SVM梯度计算（反向传播）及斯坦福线性分类Demo相关技术讲解

Hey there! Let's break down your questions step by step, using the default settings from that linear classification demo you're working with. I'll cover backward propagation calculations, how the classification line is drawn, and the pattern of line changes when you run repeated parameter updates.

1. Backward Propagation Calculation Example (Default Demo Settings) #

First, let's confirm the default setup:

• We're dealing with a 2-class linear classifier (red vs blue points) using SVM loss (the default loss function), with a margin of 1.
• Input features are 2D: each point is (x₁, x₂).
• Model parameters:
  • For red class: weight vector wᵣ = [wᵣ₁, wᵣ₂], bias bᵣ
  • For blue class: weight vector wᵦ = [wᵦ₁, wᵦ₂], bias bᵦ
• The score for a point x is:
  • sᵣ = wᵣ₁*x₁ + wᵣ₂*x₂ + bᵣ (red class score)
  • sᵦ = wᵦ₁*x₁ + wᵦ₂*x₂ + bᵦ (blue class score)

Step 1: Compute SVM Loss for a Single Sample #

Take a red point (correct class is red) as an example. The SVM loss for this sample is:

L = max(0, sᵦ - sᵣ + 1)

Let's use default initial parameters (say wᵣ = [1, 0], bᵣ = 0; wᵦ = [0, 1], bᵦ = 0) and a red point x = [3, 1]:

• sᵣ = 1*3 + 0*1 + 0 = 3
• sᵦ = 0*3 + 1*1 + 0 = 1
• L = max(0, 1 - 3 + 1) = max(0, -1) = 0 (no loss, since the correct score is higher than the wrong score by more than the margin)

Now take a misclassified red point x = [1, 2]:

• sᵣ = 1*1 + 0*2 + 0 = 1
• sᵦ = 0*1 + 1*2 + 0 = 2
• L = max(0, 2 - 1 + 1) = 2 (loss incurred because the wrong score is higher than the correct score plus margin)

Step 2: Compute Gradients (Backward Pass) #

We need to calculate the gradients of the loss L with respect to each parameter (wᵣ, wᵦ, bᵣ, bᵦ).

Case 1: Loss is 0 (correctly classified with sufficient margin)

If sᵦ - sᵣ + 1 ≤ 0, then L = 0. All gradients are 0:

• dL/dwᵣ = [0, 0], dL/dbᵣ = 0
• dL/dwᵦ = [0, 0], dL/dbᵦ = 0

Case 2: Loss > 0 (misclassified or insufficient margin)

For the misclassified red point example where L = 2:
First, apply the chain rule to find derivatives relative to scores:

• dL/dsᵣ = -1 (increasing the correct score reduces loss)
• dL/dsᵦ = 1 (increasing the wrong score increases loss)

Then compute gradients for weights and biases:

• dL/dwᵣ₁ = dL/dsᵣ * dsᵣ/dwᵣ₁ = -1 * x₁ = -1*1 = -1

• dL/dwᵣ₂ = dL/dsᵣ * dsᵣ/dwᵣ₂ = -1 * x₂ = -1*2 = -2

• dL/dbᵣ = dL/dsᵣ * dsᵣ/dbᵣ = -1 * 1 = -1

• dL/dwᵦ₁ = dL/dsᵦ * dsᵦ/dwᵦ₁ = 1 * x₁ = 1*1 = 1

• dL/dwᵦ₂ = dL/dsᵦ * dsᵦ/dwᵦ₂ = 1 * x₂ = 1*2 = 2

• dL/dbᵦ = dL/dsᵦ * dsᵦ/dbᵦ = 1 * 1 = 1

These gradients tell us how to adjust parameters to reduce loss: we subtract a small learning rate times the gradient from each parameter (gradient descent).

2. How the Classification Line is Drawn #

The classification line represents the boundary where the scores for both classes are equal: sᵣ = sᵦ. Let's rearrange this equation to get a standard line form:

Starting with:

wᵣ₁*x₁ + wᵣ₂*x₂ + bᵣ = wᵦ₁*x₁ + wᵦ₂*x₂ + bᵦ

Group like terms:

(wᵣ₁ - wᵦ₁)*x₁ + (wᵣ₂ - wᵦ₂)*x₂ + (bᵣ - bᵦ) = 0

Let w₁ = wᵣ₁ - wᵦ₁, w₂ = wᵣ₂ - wᵦ₂, b = bᵣ - bᵦ. The equation simplifies to:

w₁*x₁ + w₂*x₂ + b = 0

If we solve for x₂ (y-axis in the demo) in terms of x₁ (x-axis):

x₂ = (-w₁/w₂)*x₁ - (b/w₂)

This is the classic y = mx + c line equation, where:

• m = -w₁/w₂ (slope of the line)
• c = -b/w₂ (y-intercept)

The demo uses this equation to plot the straight line that separates the two classes.

3. Line Changes When Running "Start Repeated Parameter Update" #

When you click that button, the demo runs mini-batch gradient descent repeatedly:

• In each iteration, it calculates the loss over a subset of points, computes the gradients of the loss with respect to wᵣ, wᵦ, bᵣ, bᵦ.
• It then updates each parameter using: param = param - learning_rate * gradient

Here's how the line changes:

• Initial phase: The line moves quickly as the model fixes obvious misclassifications. It shifts and rotates to align better with the separation between red and blue points.
• Mid phase: The line's movement slows down as the loss decreases. It fine-tunes its position to minimize the total loss (for SVM, this means maximizing the margin between the closest points of each class).
• Final phase: Once the loss stops decreasing significantly, the line stabilizes. It settles at a position that best separates the two classes (or as best as a linear boundary can, if the data isn't perfectly linearly separable).

You'll notice the line doesn't jump randomly—it moves in consistent directions guided by the gradients, always trying to reduce the total classification loss.

内容的提问来源于stack exchange，提问作者Ajith