6
$\begingroup$

What is the correct number of biases in a simple convolutional layer? The question is well enough discussed, but I'm still not quite sure about that.

Say, we have (3, 32, 32)-image and apply a (32, 5, 5)-filter just like in Question about bias in Convolutional Networks

Total number of weights in the layer kernel trivially equals to $3 \times 5 \times 5 \times 32$. Now let us count biases. The link above states that total count of biases is $1 \times 32$, which makes sense because weights are shared among all output cells, so it is natural to have only one bias for each output feature map as a whole.

But from the other side: we apply activation function to each cell of output feature map separately, so if we will have different bias for each cell, they do not sum together, so the number $0 \times 0 \times 32$ instead of $1 \times 32$ makes sense too (here $0$ is the output feature map height or width).

As I can see, first approach is widely used, but I also saw the second approach in some papers.

So, ($3 \times 5 \times 5 + 1) \times 32$ or $(3 \times 5 \times 5 + 0 \times 0) \times 32$?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
7
$\begingroup$

As you say, both approaches are used. It's called tied biases if you use one bias per convolutional filter/kernel ((3x5x5 + 1)x32 overall parameters in your example) and untied biases if you use one bias per kernel and output location ((3x5x5 + OxO)x32 overall parameters in your example).

Untied biases increase the capacity of your model, so they can be a good idea if you are underfitting. But in this case using tied biases and more filters and/or layers might also help, see https://harmdevries89.wordpress.com/2015/03/27/tied-biases-vs-untied-biases/.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I think tied biases are far more common in practice (although I have not reviewed to confirm that). $\endgroup$
    – Neil Slater
    Commented Mar 20, 2017 at 11:55
  • $\begingroup$ Extremely helpful for such a beginner like me. Thanks $\endgroup$
    – Serge P.
    Commented Mar 20, 2017 at 12:43
  • $\begingroup$ Yes I also think so @NeilSlater but was not certain enough of it to write it in the answer :D :) You're welcome Serge P. $\endgroup$
    – robintibor
    Commented Mar 20, 2017 at 13:45
1
$\begingroup$

When i tried to output my CNN weights from theano's grapgh, i got one bias vector for each layer.

Improve this answer
$\endgroup$
3
  • $\begingroup$ For each feature map (channel) in each layer, I suppose. Yes, but why? That is the question. $\endgroup$
    – Serge P.
    Commented Mar 17, 2017 at 15:38
  • $\begingroup$ That was text-CNN, and there was one channel, so i can not tell you for sure, but in my opinion, there is no sence in biases for each channel in case if activation is counted from 3 channels. But also, there's no problem to apply activations to each channel and then take an avarage( so now it makes sence to sum bias to each channel) $\endgroup$
    – Ladenkov Vladislav
    Commented Mar 17, 2017 at 16:09
  • $\begingroup$ Yes, you are correct, but you talk about input channels. There is no sense in applying each bias for each input channel as they are summed up all together before apply a.f. My question is about why there is the only bias for each output channel i.e. feature map. I can add bias to each pixel of each output channel and it will make sense because this is summ-independent process (summation in each pixel is independent from others) $\endgroup$
    – Serge P.
    Commented Mar 17, 2017 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.