5
$\begingroup$

I have a convolutional layer (link) with an input 5x5x2 (width, height, depth):

The layer has 3 filters with dimensions 3x3x2, it produces an output with dimensions 3x3x3.

I have completed the forward pass:

Forward Pass

In a backward pass I calculated gradients of weights $dL/dw$, but I am not sure how to calculate $dL/dh$ (gradients for the previous layer).

Backward Pass

First or all, is my calculation of $dL/dw$ valid ? (please refer to external spread sheet linked to the question) And most importantly, how do I propagate deltas to the previous layer ?

Improve this question
$\endgroup$
10
  • $\begingroup$ Have a look at Jeremy Howard's Excel file hosted publicly on GitHub..He had simulated A CNN in EXCEL ITSELF.. $\endgroup$
    – Aditya
    Commented Mar 4, 2018 at 16:46
  • $\begingroup$ @Aditya Thank you, would you mind to share a link ? $\endgroup$
    – koryakinp
    Commented Mar 4, 2018 at 17:12
  • $\begingroup$ Here you go nbviewer.jupyter.org/github/fastai/fastai/tree/master/courses/… $\endgroup$
    – Aditya
    Commented Mar 4, 2018 at 17:21
  • $\begingroup$ @Aditya, thank you! It is very useful link, but unfortunately it does not cover backpropagation in CNN. $\endgroup$
    – koryakinp
    Commented Mar 4, 2018 at 18:16
  • $\begingroup$ What is $L$ in this question? $\endgroup$
    – JahKnows
    Commented Mar 7, 2018 at 7:11

1 Answer 1

Reset to default
1
+50
$\begingroup$

Since calculating the backward gradients that too in CNN is very math heavy and it's a bit different from normal MLP as in CNN'S we have Tensors not scalar values to multiply so People use Correlation to Calculate the Convolution Operation...

Just adding some links which I refer to from time to time thinking that someday I will be able to do this from scratch (seems impossible with deepening of Layers though...)

  • Reference 1

  • Reference 2

  • CS231N Slides (Lecture 4-6 specially, the computational graphs construction are very helpful to understand the Back-propagation Algorithm..)

It would be great if Someone can transfer the Equations into Latex here So that the question can be answered formally..

Based on the above Links, here's what I got,(Came to know that Convolutions are equivalent to Correlation)

Image 1

Image 2

Stack doesn't allow me uploading image so using gdrive..(do rotate one of the images, the first one actually)

Hope it will give you an idea...

Improve this answer
$\endgroup$
3
  • $\begingroup$ I am a student so no sir please. really liked your idea of manually calculating the gradients, Thanks $\endgroup$
    – Aditya
    Commented Mar 10, 2018 at 3:37
  • $\begingroup$ Problem is solved? $\endgroup$
    – Aditya
    Commented Mar 12, 2018 at 4:14
  • 1
    $\begingroup$ Thank you, It is much clear for me now. If you ever visit Toronto, round of beer on me. $\endgroup$
    – koryakinp
    Commented Mar 12, 2018 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.