What is the correct number of biases in a simple convolutional layer? The question is well enough discussed, but I'm still not quite sure about that.
Say, we have (3, 32, 32)-image and apply a (32, 5, 5)-filter just like in Question about bias in Convolutional Networks
Total number of weights in the layer kernel trivially equals to $3 \times 5 \times 5 \times 32$. Now let us count biases. The link above states that total count of biases is $1 \times 32$, which makes sense because weights are shared among all output cells, so it is natural to have only one bias for each output feature map as a whole.
But from the other side: we apply activation function to each cell of output feature map separately, so if we will have different bias for each cell, they do not sum together, so the number $0 \times 0 \times 32$ instead of $1 \times 32$ makes sense too (here $0$ is the output feature map height or width).
As I can see, first approach is widely used, but I also saw the second approach in some papers.
So, ($3 \times 5 \times 5 + 1) \times 32$ or $(3 \times 5 \times 5 + 0 \times 0) \times 32$?