2
$\begingroup$

I am working on CNN, and I have some doubts. Let's assume I only want one feature map, just to make things easier. And let's suppose my image is grayscale, to make things even easier. So, let's say my image is (32,32) --grayscale, hence just a channel and we don't need to write it explicitly, and my filter is (3,3) --again, one feature map, so I won't bother writing 1. I understand this will map to a (30,30) layer.

How many parameters will I have? If I understand it correctly, I will have 9 weights and one bias, so a total of 10, because we map each (3,3) subregion using the same weights. Back-propagation will determine the best values for those weights and that will give me one feature map, or a filter.

So far, so good. What I don't understand is how does the training work? I need to keep the same weights and bias when moving across the image (that's why I only have 10 parameters), but won't those change when I do back-propagation? How can I apply back-propagation and keep the same values for the weights regardless of the subregion they are applied to?

| improve this question | |
$\endgroup$
2
$\begingroup$

You are right there are just 10 params in your example.

For determining gradients, you just add up all the deltas from backpropagation in each location - i.e. you run backpropagation 30x30 = 900 times, for each position the 3x3 kernel is used, for every example in your batch (or just for one example if you are running most simple onine stochastic gradient descent), and for each position you add those delta values into a suitably-sized buffer (10 values for weight deltas, or 9 values for previous layer activation deltas). You will end up with one set of summed deltas matching your single 3x3 filter (plus a delta bias term). You then apply the summed version to update the weights of your single filter + bias.

Note this is a general rule you can apply whenever multiple gradient sources from backpropagation can be applied to any parameter - they just add. This occurs in RNNs too, or in any structure where you can set an objective function for non-output neurons.

| improve this answer | |
$\endgroup$
  • $\begingroup$ But don't you end up getting different weights if you run it 30x30 times? That is where I am getting confused. If you run backpropagation 30x30 times, how do you end up with just one filter? $\endgroup$ – user May 20 '16 at 20:31
  • $\begingroup$ No, you sum up the gradients and apply the summed version to the weights of the single filter. $\endgroup$ – Neil Slater May 20 '16 at 20:33
  • $\begingroup$ I see, so you basically calculate the gradient for all the weights, but then when you do the update you use some sort of average, like a momentum, to update the weights? $\endgroup$ – user May 21 '16 at 7:13
  • $\begingroup$ @user: Not "some sort of average". Literally just use the sum (I guess as we have a learning rate multiplier applied during weight update, it is mathematically the same thing as an average though, if that helps you visualise the problem - but you don't need any extra divisions etc). And you don't need to calculate the gradients for "all the weights" separately then combine at the end. Instead you just have one set of weight deltas which you build up by adding together all the combinations from each position. $\endgroup$ – Neil Slater May 21 '16 at 7:58
  • $\begingroup$ Although having said that, the important thing that makes a CNN work is that there is just one set of weights shared across all positions that the kernel is evaluated. How you rationalise and construct that is not important to the network generalising - however it may be important to optimisation. $\endgroup$ – Neil Slater May 21 '16 at 8:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.