7
$\begingroup$

Let's say a convolutional layer takes an input $X$ with dimensions of 5x100x100 and applies 10 filters $F$ 5x5x5, thus produces an output $O$ 10 feature maps 96x96.

During the backpropagation the layer receives $\frac{dE}{dO}$ of shape 10x96x96.

My question is how to compute $\frac{dE}{dF}$ ?

According to that article $\frac{dE}{dF}$ can be calculated as convolution between $X$ and $\frac{dE}{dO}$. Unfortunately, the article does not cover a case with multiple filters and multiple input channels.

Since $X$ has shape 5x100x100 and $\frac{dE}{dO}$ has shape 10X96x96 the depth of $X$ equals to 5 and the depth of $\frac{dE}{dO}$ equals to 10. So the depth dimension does not match. How to compute convolution in that case ?

Improve this question
$\endgroup$
2
  • $\begingroup$ Multiple filters are not a problem, because we can calculate their gradient separately. However, the number of input channels does seem to be a problem, because these channels collapse when we calculate $O$. You could get the shapes to line up if you broadcast $\frac{dE}{dO}$ to 5 channels, so if there is a solution I think this must be it. However I'm not sure if this works, because information is lost when you collapse the input and filter channels into calculating $O$, so maybe there isn't a nice analogy for doing backprop via convolution with multiple channels. $\endgroup$
    – Imran
    Oct 1, 2018 at 21:07
  • $\begingroup$ One note: It's OK that the 100x100 shape of one channel of $X$ does not match the 96x96 shape of $\frac{dE}{dO}$, because the output shape of a full convolution will take the shape of the left argument, since the right side slides all the way across until only one row or col overlaps in each dimension - so the number of positions is the same as the shape of $X$. However, there is still a mismatch in channels, ie 5 vs 1 for each filter (not vs 10 as you have, which is the number of feature planes). $\endgroup$
    – Imran
    Oct 1, 2018 at 21:12

2 Answers 2

Reset to default
1
$\begingroup$

One possible option is padding the lower dimensional object to match the size of the higher dimensional object. Padding is most commonly done so the input and output volume will always have the same spatial dimensions. The additional cells could be padded with any value. It is most common to use zero or the mean of the other cells, depending on the filter operation.

Improve this answer
$\endgroup$
1
$\begingroup$

I did a little bit of research and found that in order to compute $\frac{dE}{dF}$ I need to do convolution between input ${X}$ and tiled $\frac{dE}{dO}$.

Hope that little illustration helps:

enter image description here

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.