18
$\begingroup$

I am trying to figure out how many weights and biases are needed for CNN.

Say I have a (3, 32, 32)-image and want to apply a (32, 5, 5)-filter. For each feature map I have 5x5 weights, so I should have 3 x (5x5) x 32 parameters. Now I need to add the bias. I believe I only have (3 x (5x5) + 1) x 32 parameters, so is the bias the same across all colors (RGB)?

Is this correct? Do I keep the same bias for each image across its depth (in this case 3) while I use different weights? Why is that?

$\endgroup$

4 Answers 4

12
$\begingroup$

Bias operates per virtual neuron, so there is no value in having multiple bias inputs where there is a single output - that would equivalent to just adding up the different bias weights into a single bias.

In the feature maps that are the output of the first hidden layer, the colours are no longer kept separate*. Effectively each feature map is a "channel" in the next layer, although they are usually visualised separately where the input is visualised with channels combined. Another way of thinking about this is that the separate RGB channels in the original image are 3 "feature maps" in the input.

It doesn't matter how many channels or features are in a previous layer, the output to each feature map in the next layer is a single value in that map. One output value corresponds to a single virtual neuron, needing one bias weight.

In a CNN, as you explain in the question, the same weights (including bias weight) are shared at each point in the output feature map. So each feature map has its own bias weight as well as previous_layer_num_features x kernel_width x kernel_height connection weights.

So yes, your example resulting in (3 x (5x5) + 1) x 32 weights total for the first layer is correct for a CNN with first hidden layer processing RGB input into 32 separate feature maps.


* You may be getting confused by seeing visualisation of CNN weights which can be separated into the colour channels that they operate on.

$\endgroup$
3
  • $\begingroup$ I thought there was one bias per filter...do we have different definitions of virtual neurons? $\endgroup$ Mar 6, 2018 at 21:42
  • $\begingroup$ @CharlieParker In terms of number of parameters, one bias per filter is same as one bias per neuron. There are a few equivalent ways to view the architecture of a CNN feature map. One would be as the "fully expanded" layer with large numbers of neurons all sharing the same parameters. Another would be to think of each convolutional filter as describing a single neuron's local connections, and the layer repeatedly uses this same neuron. So I think we probably have the same definition of a neuron, but the answer describes things from just one perspective. $\endgroup$ Mar 6, 2018 at 23:00
  • $\begingroup$ Interesting, I guess you think of each neuron for each filter, I was thinking for some reason each activation for each neuron, but the filter activates in many places in the image so of course I had many more neurons in mind. $\endgroup$ Mar 6, 2018 at 23:04
2
$\begingroup$

here an easy example:

If your input is I X I X C e.g a picture with I X I = 10 x 10 pixels, grayscale so you have only 1 Channel = C results in input = 10 X 10 x 1. And your filter is F X F and you apply K Filters e.g 3 X 3 X 10 Filters then your output is (F X F X C +1) X K where +1 means you have 1 bias per filter. (3 X 3 X 1 + 1) X 10 = 100 parameters to train. If you have RGB then you have (3 X 3 X 3 + 1) X 10 = 280 parameters to train. For every channel, you have another filter. And every filter has one bias.

Here to check this again after reading:

Input I x I x C
Filter F x F (x K) // K times applied
Parameters (F x F x C + 1) x K // where +1 bias per filter, and K is the number of filters

Now check this awesome cheat sheet from Stanford university here

$\endgroup$
2
$\begingroup$

Since in CNN, we are taking one filter to indicate one feature. We introduce a variable(b) to incorporate the bias from that particular filter. Hence, each filter takes into account the bias that it can cause. The bias is not due to individual filter weights but the whole filter itself. I hope that answers your question.

$\endgroup$
1
$\begingroup$

It is the property of CNNs that they use shared weights and biases(same weights and bias for all the hidden neurons in a layer) in order to detect the same feature. This leads to a more deep learning as compared to simple neural networks. You can read this out as a reference :

http://deeplearning.net/tutorial/lenet.html
http://neuralnetworksanddeeplearning.com/chap6.html#introducing_convolutional_networks

$\endgroup$
1
  • $\begingroup$ I am asking a different question. My question is about the bias. From the formula, it looks that for each feature map there are different weights, but the same bias is used across all the feature maps. $\endgroup$
    – user
    May 21, 2016 at 5:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.