0
$\begingroup$

I understand that the 3 main layers for CNN are convolutional layer, ReLU layer and pooling layer.

However, I do not understand how CNN updates its weights and biases using backpropagation.

I understand that backpropagation uses partial derivatives. But I do not see how they are used in CNN.

Any reference is appreciated.

$\endgroup$
1
$\begingroup$

the 3 main layers for CNN are convolutional layer, ReLU layer and pooling layer

Not necessarily in fact. Conv layers are the only layers that you absolutely need in order to implement a CNN. All the other are not strictly necessary. For example, I trained a CNN on Fashion MNIST dataset using ELU activation (you can check my Notebook here). Moreover, if the size of the input data is not too big and you have enough computing power, you can get away with Pooling layers and use only the Convolutional ones. (As computing power increasese, Pool layers will be progressively abandoned since they loose too much information).

Coming to your question:

how CNN updates its weights and biases using backpropagation. I understand that backpropagation uses partial derivatives. But I do not see how they are used in CNN.

Backpropagation works this way: it calculates how much each weight is contributing to the final Loss value (this computation is done finding the first partial derivative with respect to each weight). This trick works in the same way for "normal", Dense layers as for Convolutional layers. The only difference is that Conv layers are not fully connected, i.e. not all the nodes of one layer are connected with all the nodes of the following layer. The difference is that only the nodes that fall under each sliding filter are connected. That is a trick to process large amount of data (such as pixels).

Conv layers have a different architecture from Dense layers, but their training works exactly the same as for Dense layers.

Pool layers, on the other side, do not contain any learnable parameter. They are just mechanical operations that don't require any training.


EDIT: This is a great explanation of how backprop works.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I cannot find any notebook in your answer. $\endgroup$ – Idonknow Sep 30 '19 at 7:35
  • $\begingroup$ May I know what are some equations required for backpropagate CNN? $\endgroup$ – Idonknow Sep 30 '19 at 7:36
  • $\begingroup$ If possible, can you provide some references or give some outlines or equations on how backpropagation in CNN is carried out? $\endgroup$ – Idonknow Sep 30 '19 at 7:40
  • $\begingroup$ Sorry I forgot the link. I edited the answer above and added a reference to a great blog post on backprop $\endgroup$ – Leevo Sep 30 '19 at 7:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.