What is the correct number of biases in a simple convolutional layer? The question is well enough discussed, but I'm still not quite sure about that.

Say, we have (3, 32, 32)-image and apply a (32, 5, 5)-filter just like in Question about bias in Convolutional Networks

Total number of weights in the layer kernel trivially equals to $3x5x5x32$. Now let us count biases. The link above states that total count of biases is $1x32$, which makes sense because weights are shared among all output cells, so it is natural to have only one bias for each output feature map as a whole.

But from the other side: we apply activation function to each cell of output feature map separately, so if we will have different bias for each cell, they do not sum together, so the number $0x0x32$ instead of $1x32$ makes sense too (here 0 is the output feature map height or width).

As I can see, first approach is widely used, but I also saw the second approach in some papers.

So, ($3x5x5 + 1)x32$ or $(3x5x5 + 0x0)x32$?


As you say, both approaches are used. It's called tied biases if you use one bias per convolutional filter/kernel ((3x5x5 + 1)x32 overall parameters in your example) and untied biases if you use one bias per kernel and output location ((3x5x5 + OxO)x32 overall parameters in your example).

Untied biases increase the capacity of your model, so they can be a good idea if you are underfitting. But in this case using tied biases and more filters and/or layers might also help, see https://harmdevries89.wordpress.com/2015/03/27/tied-biases-vs-untied-biases/.

  • $\begingroup$ I think tied biases are far more common in practice (although I have not reviewed to confirm that). $\endgroup$ – Neil Slater Mar 20 '17 at 11:55
  • $\begingroup$ Extremely helpful for such a beginner like me. Thanks $\endgroup$ – Serge P. Mar 20 '17 at 12:43
  • $\begingroup$ Yes I also think so @NeilSlater but was not certain enough of it to write it in the answer :D :) You're welcome Serge P. $\endgroup$ – robintibor Mar 20 '17 at 13:45

When i tried to output my CNN weights from theano's grapgh, i got one bias vector for each layer.

  • $\begingroup$ For each feature map (channel) in each layer, I suppose. Yes, but why? That is the question. $\endgroup$ – Serge P. Mar 17 '17 at 15:38
  • $\begingroup$ That was text-CNN, and there was one channel, so i can not tell you for sure, but in my opinion, there is no sence in biases for each channel in case if activation is counted from 3 channels. But also, there's no problem to apply activations to each channel and then take an avarage( so now it makes sence to sum bias to each channel) $\endgroup$ – Ladenkov Vladislav Mar 17 '17 at 16:09
  • $\begingroup$ Yes, you are correct, but you talk about input channels. There is no sense in applying each bias for each input channel as they are summed up all together before apply a.f. My question is about why there is the only bias for each output channel i.e. feature map. I can add bias to each pixel of each output channel and it will make sense because this is summ-independent process (summation in each pixel is independent from others) $\endgroup$ – Serge P. Mar 17 '17 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.