梯度检查中crossed kink及max函数判定法相关技术疑问

Hey there, let's break down these questions step by step—this is a common gotcha when doing gradient checks with non-smooth functions like max()!

1. 这里的“identities”指什么？ #

When we talk about the "identities" in a max(x, y) function, we're referring to which of the two inputs was selected as the maximum value during forward propagation. For example:

• If x > y, the identity is "x" (it's the "winner" that becomes the output of the max function)
• If y > x, the identity is "y"

We track these identities because the max() function is non-smooth at the point where x = y (this is the "kink"). The gradient of max(x,y) depends entirely on which input was the winner—so if the winner changes when we perturb the parameters (for gradient checking), our numerical gradient will be off.

2. 梯度检查中的x、y代表什么？ #

First, let's clarify terminology to avoid confusion:

• The x and y in max(x, y) are the two input values fed into the max function during your model's forward pass. These could be intermediate activations (e.g., outputs of linear layers) or even direct model parameters, depending on where the max operation lives in your network.
• The x mentioned in gradient check context (like f(x+h) and f(x−h)) refers to the vector of model parameters you're trying to compute the gradient for. h is a tiny perturbation (usually ~1e-5) we add/subtract to estimate the numerical gradient.

3. 具体示例帮助理解 #

Let's use a super simple scenario to make this concrete:

Suppose our loss function is L = max(a, b), where:

• a = w * 1 + 2 (w is our only model parameter, input is fixed at 1)
• b = 3 (a constant value)

Case 1: No crossed kink (numerical gradient is accurate) #

Current parameter value: w = 2

• Forward pass: a = 2*1 +2 =4, b=3 → max(4,3)=4, identity is "a"
• Compute f(w+h): w+h=2.00001 → a=4.00001, max=4.00001 (identity still "a")
• Compute f(w-h): w-h=1.99999 → a=3.99999, max=3.99999 (identity still "a")
• Numerical gradient: (4.00001 - 3.99999)/(2*1e-5) = 0.00002/2e-5 = 1
• Analytic gradient: Since a > b, the gradient of L with respect to w is 1 (matches numerical gradient perfectly)

Case 2: Crossed kink (numerical gradient is inaccurate) #

Current parameter value: w =1

• Forward pass: a=1*1+2=3, b=3 → max(3,3)=3 (we might arbitrarily pick "a" as the identity here)
• Compute f(w+h): w+h=1.00001 → a=3.00001, max=3.00001 (identity switches to "a")
• Compute f(w-h): w-h=0.99999 → a=2.99999, max=3 (identity switches to "b")
• Numerical gradient: (3.00001 -3)/(2*1e-5) = 0.00001/2e-5 = 0.5
• Analytic gradient: If we stuck with the forward pass identity "a", the analytic gradient would be 1. The numerical gradient is wrong here because we crossed the kink (where a=b) when perturbing w—changing the winner identity in one of the perturbed calculations.

This is exactly the scenario cs231n is warning about: when perturbing parameters causes the winner of a max function to change, your numerical gradient won't match the analytic one, and you shouldn't trust the gradient check result in this case.

内容的提问来源于stack exchange，提问作者alec.tu