I have a convolutional layer (link) with an input 5x5x2 (width, height, depth):

The layer has 3 filters with dimensions 3x3x2, it produces an output with dimensions 3x3x3.

I have completed the forward pass:

Forward Pass

In a backward pass I calculated gradients of weights $dL/dw$, but I am not sure how to calculate $dL/dh$ (gradients for the previous layer).

Backward Pass

First or all, is my calculation of $dL/dw$ valid ? (please refer to external spread sheet linked to the question) And most importantly, how do I propagate deltas to the previous layer ?

  • $\begingroup$ Have a look at Jeremy Howard's Excel file hosted publicly on GitHub..He had simulated A CNN in EXCEL ITSELF.. $\endgroup$ – Aditya Mar 4 '18 at 16:46
  • $\begingroup$ @Aditya Thank you, would you mind to share a link ? $\endgroup$ – koryakinp Mar 4 '18 at 17:12
  • $\begingroup$ Here you go nbviewer.jupyter.org/github/fastai/fastai/tree/master/courses/… $\endgroup$ – Aditya Mar 4 '18 at 17:21
  • $\begingroup$ @Aditya, thank you! It is very useful link, but unfortunately it does not cover backpropagation in CNN. $\endgroup$ – koryakinp Mar 4 '18 at 18:16
  • $\begingroup$ What is $L$ in this question? $\endgroup$ – JahKnows Mar 7 '18 at 7:11

Since calculating the backward gradients that too in CNN is very math heavy and it's a bit different from normal MLP as in CNN'S we have Tensors not scalar values to multiply so People use Correlation to Calculate the Convolution Operation...

Just adding some links which I refer to from time to time thinking that someday I will be able to do this from scratch (seems impossible with deepening of Layers though...)

  • Reference 1

  • Reference 2

  • CS231N Slides (Lecture 4-6 specially, the computational graphs construction are very helpful to understand the Back-propagation Algorithm..)

It would be great if Someone can transfer the Equations into Latex here So that the question can be answered formally..

Based on the above Links, here's what I got,(Came to know that Convolutions are equivalent to Correlation)

Image 1

Image 2

Stack doesn't allow me uploading image so using gdrive..(do rotate one of the images, the first one actually)

Hope it will give you an idea...

|improve this answer|||||
  • $\begingroup$ I am a student so no sir please. really liked your idea of manually calculating the gradients, Thanks $\endgroup$ – Aditya Mar 10 '18 at 3:37
  • $\begingroup$ Problem is solved? $\endgroup$ – Aditya Mar 12 '18 at 4:14
  • 1
    $\begingroup$ Thank you, It is much clear for me now. If you ever visit Toronto, round of beer on me. $\endgroup$ – koryakinp Mar 12 '18 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.