8
$\begingroup$

Consider a Convolutional Neural Network (CNN) for image classification. In order to detect local features, weight-sharing is used among units in the same convolutional layer. In such a network, the kernel weights are updated via the backpropagation algorithm.

An update for the kernel weight $h_j$ in layer $l$ would be as follows:

$h_j^l = h_j^l - \eta \cdot \frac{\delta R}{\delta h_j^l} = h_j^l - \eta \cdot \frac{\delta R}{\delta x_j^{L}} \cdot \frac{\delta x_j^{L}}{\delta x_j^{L - 1}} \cdot ... \cdot \frac{\delta x_j^{l}}{\delta h_j^l}$

How can the kernel weights be updated and still be the same (=shared)?

I have 2 possible explanations:

  1. Weights of the same layer, which are initialized to the same value, will stay the same (independently of the input). This would imply that the expression $\frac{\delta R}{\delta h_j^l}$ is the same for all of these weights $h_1^l$ to $h_J^l$. This does not make sense, since $x_j^l$ is different for different j's. Or am I missing something here?

  2. There is a trick, e.g. after the back-propagation update, the shared weights are set to their mean.

EDIT The confusion I had was that I didn't consider that if a weight is shared, its parameter $h_j^l$ appears several times in the loss function. When differentiating for $h_j^l$, several terms (considering the according inputs) will "survive". Therefore the updates will be the same.

$\endgroup$
6
$\begingroup$

I think you're misunderstanding what "weight sharing" means here. A convolutional layer is generally comprised of many "filters", which are usually 2x2 or 3x3. These filters are applied in a "sliding window" across the entire layer's input. The "weight sharing" is using fixed weights for this filter across the entire input. It does not mean that all of the filters are equivalent.

To be concrete, let's imagine a 2x2 filter $F$ striding a 3x3 input $X$ with padding, so the filter gets applied 4 times. Let's denote the unrolled filter $\beta$.

$$X = \begin{bmatrix} x_{11} & x_{12} & x_{13} \\ x_{21} & x_{22} & x_{23} \\ x_{31} & x_{32} & x_{33} \end{bmatrix}$$

$$F = \begin{bmatrix} w_{11} & w_{12} \\ w_{21} & w_{22} \end{bmatrix}$$

$$\beta= [w_{11}, w_{12}, w_{21}, w_{22} ]$$

$$F*X = \begin{bmatrix} \beta \cdot [x_{11}, x_{12}, x_{21}, x_{22}] & \beta \cdot [x_{12}, x_{13}, x_{22}, x_{23}] \\ \beta \cdot [x_{21}, x_{22}, x_{31}, x_{32}] & \beta \cdot [x_{22}, x_{23}, x_{32}, x_{33}] \end{bmatrix} $$

"Weight sharing" means when we apply this 2x2 filter to our 3x3 input, we reuse the same four weights given by the filter across the entire input. The alternative would be each filter application having its own set of inputs (which would really be a separate filter for each region of the image), giving a total of 16 weights, or a dense layer with 4 nodes giving 36 weights.

Sharing weights in this way significantly reduces the number of weights we have to learn, making it easier to learn very deep architectures, and additionally allows us to learn features that are agnostic to what region of the input is being considered.

EDIT: To further motivate this, here's an animation of a 3x3 filter applied to a 5x5 input

enter image description here

$\endgroup$
  • 1
    $\begingroup$ Thank you, I can now boil it down to a simpler question: Are the weights represented by $\beta$ learned via back-propagation? $\endgroup$ – Andi R Jan 18 '18 at 8:55
  • $\begingroup$ Yup! They sure are. $\endgroup$ – David Marx Jan 18 '18 at 9:45
  • 1
    $\begingroup$ How can $\beta$ be learned and be the same (i.e. shared) across the entire input? Since the back-propagation algorithm also considers the input, for 2 different inputs, the $\beta$ should be updated differently. $\endgroup$ – Andi R Jan 18 '18 at 10:17
  • 1
    $\begingroup$ The partial derivatives in the weight update are calculated relative to the weights ($\beta$), not the input. It's really no different from backprop in a MLP. $\endgroup$ – David Marx Jan 18 '18 at 10:51
  • $\begingroup$ I agree with David here, you are confusing input with weights convolutions are simple operations where a kernel is applied on a input image as shown above and using backprop the kernel weights are updated such that they minimize the loss function.First loss is calculated w.r.t to your activation * rate of change of actiavtion w.r.t to weighted sum of input * rate of change of weighted sum of inputs w.r.t rate of change of weights(kernel weights here) . $\endgroup$ – khwaja wisal Aug 26 at 19:33
5
$\begingroup$

I'm not sure if you can change accepted answers, but since the only answer to your question on back propagation is one about forward propagation, I decided to give it a go.

Essentially, you treat the weight delta ($\frac{\delta R}{\delta h_j^l}$) the same as you would a weight delta for a linear neuron, but train it once for each time you overlaid your filter (kernel) on the input, all in a single backprop pass. The result is the sum of the deltas for all overlays of your filter. I think in your notation, this would be $\frac{\delta R}{\delta h_j^l} = \sum_{i=1}^nx_i^{l-1}\frac{\delta R}{\delta x_j^{l+1}}$ where $x^{l-1}$ is an input that was multiplied with $h_j^l$ for one of your overlays during forward propagation, and $x^{l+1}$ is the output that resulted from that overlay.

Between the two results of backprop through a convolutional layer (parameter deltas, and input deltas) it sounds like you're more interested in parameter deltas, or more specifically, the weight matrix deltas ($\frac{\delta R}{\delta h_j^l}$). For the sake of completeness, I'll go over both, with the following as our example layer:

If you have a 1D set of inputs $[1, 2, 3]$, and a filter $[0.3, 0.5]$ that is applied with stride 1, no padding, zero bias, and no activation function, then the activation of your filter would have looked like $[1*0.3+2*0.5, 2*0.3+3*0.5] = [1.3, 2.1]$. When you come back through this layer on your backprop pass, let's say the activation deltas you use for your calculations are $[-0.1, 0.2]$.

Weight deltas:

When you came through in the forward pass, you cached your inputs $[1, 2, 3]$, let's call that A_prev, since it's likely the activation of your previous layer. For each potential overlay of your filter (in this case, you can only overlay it on the input in two places [1,2,3] and [1,2,3]), take that slice of the input A_slice, multiply each element by the associated output delta dZ, and add it to your weight delta dW for this pass. In this example, you would add $[1*-0.1, 2*-0.1]$ to dW for the first overlay, then add $[2*0.2, 3*0.2]$ for the second overlay. All told, your dW for this convolutional layer on this backprop pass is $[0.3, 0.4]$.

Bias deltas:

Same as for weight deltas, but just add your output delta without multiplying by the input matrix.

Input deltas:

Reconstruct the shape of the input for this layer, call it dA_prev, initialize it to zeros, and walk through the configurations in which you overlaid your filter on the input. For each overlay, multiply your weight matrix by the output delta associated with this overlay, and add that to the slice of dA_prev associated with this overlay. That is to say, overlay 1 will add $[0.3 * -0.1, 0.5 * -0.1] = [-0.03, -0.05]$ to dA_prev resulting in $[-0.03, -0.05, 0]$, then overlay 2 will add $[0.3 * 0.2, 0.5 * 0.2] = [0.06, 0.1]$, resulting in $[-0.03, 0.01, 0.1]$ for dA_prev.

This is a pretty good source if you want to read the same answer in different terms: Link

$\endgroup$
  • $\begingroup$ Thanks, this actually answers the original question... $\endgroup$ – Yan King Yin Jun 3 at 14:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.