CNN - How does backpropagation with weight-sharing work exactly?

Question

Consider a Convolutional Neural Network (CNN) for image classification. In order to detect local features, weight-sharing is used among units in the same convolutional layer. In such a network, the kernel weights are updated via the backpropagation algorithm.

An update for the kernel weight $h_j$ in layer $l$ would be as follows:

$h_j^l = h_j^l - \eta \cdot \frac{\delta R}{\delta h_j^l} = h_j^l - \eta \cdot \frac{\delta R}{\delta x_j^{L}} \cdot \frac{\delta x_j^{L}}{\delta x_j^{L - 1}} \cdot ... \cdot \frac{\delta x_j^{l}}{\delta h_j^l}$

How can the kernel weights be updated and still be the same (=shared)?

I have 2 possible explanations:

1. Weights of the same layer, which are initialized to the same value, will stay the same (independently of the input). This would imply that the expression $\frac{\delta R}{\delta h_j^l}$ is the same for all of these weights $h_1^l$ to $h_J^l$. This does not make sense, since $x_j^l$ is different for different j's. Or am I missing something here?

2. There is a trick, e.g. after the back-propagation update, the shared weights are set to their mean.

EDIT The confusion I had was that I didn't consider that if a weight is shared, its parameter $h_j^l$ appears several times in the loss function. When differentiating for $h_j^l$, several terms (considering the according inputs) will "survive". Therefore the updates will be the same.

Answer 1

I think you're misunderstanding what "weight sharing" means here. A convolutional layer is generally comprised of many "filters", which are usually 2x2 or 3x3. These filters are applied in a "sliding window" across the entire layer's input. The "weight sharing" is using fixed weights for this filter across the entire input. It does not mean that all of the filters are equivalent.

To be concrete, let's imagine a 2x2 filter $F$ striding a 3x3 input $X$ with padding, so the filter gets applied 4 times. Let's denote the unrolled filter $\beta$.

$$X = \begin{bmatrix} x_{11} & x_{12} & x_{13} \\ x_{21} & x_{22} & x_{23} \\ x_{31} & x_{32} & x_{33} \end{bmatrix}$$

$$F = \begin{bmatrix} w_{11} & w_{12} \\ w_{21} & w_{22} \end{bmatrix}$$

$$\beta= [w_{11}, w_{12}, w_{21}, w_{22} ]$$

$$F*X = \begin{bmatrix} \beta \cdot [x_{11}, x_{12}, x_{21}, x_{22}] & \beta \cdot [x_{12}, x_{13}, x_{22}, x_{23}] \\ \beta \cdot [x_{21}, x_{22}, x_{31}, x_{32}] & \beta \cdot [x_{22}, x_{23}, x_{32}, x_{33}] \end{bmatrix} $$

"Weight sharing" means when we apply this 2x2 filter to our 3x3 input, we reuse the same four weights given by the filter across the entire input. The alternative would be each filter application having its own set of inputs (which would really be a separate filter for each region of the image), giving a total of 16 weights, or a dense layer with 4 nodes giving 36 weights.

Sharing weights in this way significantly reduces the number of weights we have to learn, making it easier to learn very deep architectures, and additionally allows us to learn features that are agnostic to what region of the input is being considered.

EDIT: To further motivate this, here's an animation of a 3x3 filter applied to a 5x5 input

Answer 2

I'm not sure if you can change accepted answers, but since the only answer to your question on back propagation is one about forward propagation, I decided to give it a go.

Essentially, you treat the weight delta ($\frac{\delta R}{\delta h_j^l}$) the same as you would a weight delta for a linear neuron, but train it once for each time you overlaid your filter (kernel) on the input, all in a single backprop pass. The result is the sum of the deltas for all overlays of your filter. I think in your notation, this would be $\frac{\delta R}{\delta h_j^l} = \sum_{i=1}^nx_i^{l-1}\frac{\delta R}{\delta x_j^{l+1}}$ where $x^{l-1}$ is an input that was multiplied with $h_j^l$ for one of your overlays during forward propagation, and $x^{l+1}$ is the output that resulted from that overlay.

Between the two results of backprop through a convolutional layer (parameter deltas, and input deltas) it sounds like you're more interested in parameter deltas, or more specifically, the weight matrix deltas ($\frac{\delta R}{\delta h_j^l}$). For the sake of completeness, I'll go over both, with the following as our example layer:

If you have a 1D set of inputs $[1, 2, 3]$, and a filter $[0.3, 0.5]$ that is applied with stride 1, no padding, zero bias, and no activation function, then the activation of your filter would have looked like $[1*0.3+2*0.5, 2*0.3+3*0.5] = [1.3, 2.1]$. When you come back through this layer on your backprop pass, let's say the activation deltas you use for your calculations are $[-0.1, 0.2]$.

Weight deltas:

When you came through in the forward pass, you cached your inputs $[1, 2, 3]$, let's call that A_prev, since it's likely the activation of your previous layer. For each potential overlay of your filter (in this case, you can only overlay it on the input in two places [1,2,3] and [1,2,3]), take that slice of the input A_slice, multiply each element by the associated output delta dZ, and add it to your weight delta dW for this pass. In this example, you would add $[1*-0.1, 2*-0.1]$ to dW for the first overlay, then add $[2*0.2, 3*0.2]$ for the second overlay. All told, your dW for this convolutional layer on this backprop pass is $[0.3, 0.4]$.

Bias deltas:

Same as for weight deltas, but just add your output delta without multiplying by the input matrix.

Input deltas:

Reconstruct the shape of the input for this layer, call it dA_prev, initialize it to zeros, and walk through the configurations in which you overlaid your filter on the input. For each overlay, multiply your weight matrix by the output delta associated with this overlay, and add that to the slice of dA_prev associated with this overlay. That is to say, overlay 1 will add $[0.3 * -0.1, 0.5 * -0.1] = [-0.03, -0.05]$ to dA_prev resulting in $[-0.03, -0.05, 0]$, then overlay 2 will add $[0.3 * 0.2, 0.5 * 0.2] = [0.06, 0.1]$, resulting in $[-0.03, 0.01, 0.1]$ for dA_prev.

This is a pretty good source if you want to read the same answer in different terms: Link