求解卷积层损失对权重的偏导及指定损失下的梯度计算

Alright, let's break down these two gradient computation tasks step by step, with concrete formulas and intuitive explanations:

1. Gradient of Loss with Respect to Convolution Layer Weights #

First, let's recap the forward pass of a convolution layer to set the stage:

• Let the 4D input tensor be $x \in \mathbb{R}^{B \times H_x \times W_x \times D_x}$ (B = batch size, $H_x/W_x$ = input height/width, $D_x$ = input channels)
• Convolution weights $W \in \mathbb{R}^{H_k \times W_k \times D_x \times D_o}$ ($H_k/W_k$ = kernel size, $D_o$ = output channels)
• Output feature map $z_{conv} \in \mathbb{R}^{B \times H_o \times W_o \times D_o}$ (with stride=1; $H_o/W_o$ = output dimensions dependent on padding)

The forward pass for a single output element is:
$$z_{conv}[b, h, w, d] = \sum_{i=0}^{{H_k-1}\sum_{j=0}}{W_k-1}\sum_{c=0}^{D_x-1} x[b, h+i, w+j, c] \cdot W[i,j,c,d] + b[d]$$

Using the chain rule, the gradient of the loss with respect to a weight element $W[i,j,c,d]$ is the sum over all batch samples and output positions of the product of:

• The loss gradient with respect to the corresponding output element $\frac{\partial \text{Loss}}{\partial z_{conv}[b,h,w,d]}$
• The input element that $W[i,j,c,d]$ multiplies in the forward pass ($x[b, h+i, w+j, c]$)

Mathematically, this is:
$$\frac{\partial \text{Loss}}{\partial W[i,j,c,d]} = \sum_{b=0}^{{B-1}\sum_{h=0}}{H_o-1}\sum_{w=0}^{W_o-1} \frac{\partial \text{Loss}}{\partial z_{conv}[b,h,w,d]} \cdot x[b, h+i, w+j, c]$$

In practice, this is equivalent to performing a cross-correlation between the input tensor $x$ and the loss gradient tensor $\frac{\partial \text{Loss}}{\partial z_{conv}}$.

2. Gradients for Cross-Entropy Loss #

First, let's derive the gradient of the given cross-entropy loss with respect to the softmax input $z$ (10-dimensional vector per sample):

The loss function is:
$$\mathrm{loss} = \sum_{i=1}^N \left( -z_{i}[y_i] + \log{\left( \sum_{c=1}^{10} \exp(z_{i}[c]) \right)} \right)$$
where $N$ is the number of samples, $y_i$ is the true class label for sample $i$, and $z_i[c]$ is the $c$-th element of the softmax input vector for sample $i$.

Step 1: Gradient of Loss w.r.t. Softmax Input $z$ #

For a single sample $i$, let $p_i[c] = \frac{\exp(z_i[c])}{\sum_{k=1}^{10}\exp(z_i[k])}$ (the softmax output for class $c$). The gradient $\frac{\partial L_i}{\partial z_i[c]}$ (where $L_i$ is the loss for sample $i$) is:

• If $c = y_i$: $\frac{\partial L_i}{\partial z_i[c]} = p_i[c] - 1$
• If $c \neq y_i$: $\frac{\partial L_i}{\partial z_i[c]} = p_i[c]$

We can write this concisely using an indicator function:
$$\frac{\partial \text{Loss}}{\partial z_i[c]} = p_i[c] - \mathbb{I}(c = y_i)$$
where $\mathbb{I}(\cdot)$ returns 1 if the condition is true, 0 otherwise.

Step 2: Connect to Convolution Layer Output #

Assume the softmax input $z$ is derived from the convolution output $z_{conv}$ (e.g., via global average pooling, flattening + fully connected layer, or a 1x1 convolution that outputs 10 channels). For simplicity, let's use global average pooling (GAP) as an example:
$$z_i[c] = \frac{1}{H_o W_o} \sum_{h=0}^{{H_o-1}\sum_{w=0}}{W_o-1} z_{conv}[i, h, w, c]$$

The gradient of the loss with respect to the convolution output element $\delta_{conv}[i,h,w,c] = \frac{\partial \text{Loss}}{\partial z_{conv}[i,h,w,c]}$ is then:
$$\delta_{conv}[i,h,w,c] = \frac{1}{H_o W_o} \cdot \left( p_i[c] - \mathbb{I}(c = y_i) \right)$$

Step 3: Gradient of Loss w.r.t. Convolution Weights $W$ #

Using the result from Section 1, substitute $\delta_{conv}$ into the weight gradient formula:
$$\frac{\partial \text{Loss}}{\partial W[i,j,c,d]} = \sum_{b=0}^{{B-1}\sum_{h=0}}{H_o-1}\sum_{w=0}^{W_o-1} \delta_{conv}[b,h,w,d] \cdot x[b, h+i, w+j, c]$$

Step 4: Gradient of Loss w.r.t. 4D Input $x$ #

To compute $\frac{\partial \text{Loss}}{\partial x[b,h,w,c]}$, we need to propagate the gradient back from the convolution output to the input. This is equivalent to performing a transposed convolution (or full convolution) of $\delta_{conv}$ with the kernel weights $W$, accounting for boundary valid positions.

Mathematically:
$$\frac{\partial \text{Loss}}{\partial x[b,h,w,c]} = \sum_{i=0}^{{H_k-1}\sum_{j=0}}{W_k-1}\sum_{d=0}^{D_o-1} \mathbb{I}(h-i \in [0, H_o-1], w-j \in [0, W_o-1]) \cdot \delta_{conv}[b, h-i, w-j, d] \cdot W[i,j,c,d]$$

The indicator function ensures we only sum over valid output positions that could have contributed to the input position $(b,h,w,c)$ during the forward pass.

内容的提问来源于stack exchange，提问作者walruses_boI_984