Understanding how many biases are there in a neural network?

Question

I am trying to understand biases in neural nets, but different websites show very different answers.

For example, how many biases is there in a fully connected neural network with a single input layer with 5 units and a single output layer with 4 units? And what about a fully connected neural network with a single input layer with 5 units, a single hidden layer with 4 units, and a single output layer with 3 units?

For example, if I understand this correctly, https://ai.stackexchange.com/questions/17584/why-does-the-bias-need-to-be-a-vector-in-a-neural-network, the answer of the first should be 5 and for the second 4 + 3. Each neuron except for in the input-layer has a bias.

However, at https://ayearofai.com/rohan-5-what-are-bias-units-828d942b4f52, it is explained such that each layer including the input-layer has one bias. So the answer to the example above is one in the first and two in the second.

What is correct? What am I misunderstanding here?

Answer 1

The sources are both correct, they implement bias in different ways, and are counting slightly different things:

• Your first source implements a separate bias vector in each output layer in addition to the weights, and is referring to the dimension of the bias vector when it is counting biases.

• Your second source implements bias as both a separate fixed value $1.0$ in each input layer, and a larger weights matrix with an extra column containing the actual learned bias value. It is referring to the extra added value in each layer when counting the "biases" - more accurately it is counting the added bias "signals" and not the learned bias values, because the learned bias values are implemented inside the weights matrix when using this approach.

In both cases, the number of learned values added due to bias are the same, and are the same as this:

the answer of the first should be 5 and for the second 4 + 3. Each neuron except for in the input-layer has a bias

In the second case the equivalent values appear in weight matrices as extra columns. It's really just an implementation difference. By adding a fixed bias signal to inputs, it simplifies the learning update for each layer to occur to only one matrix, as opposed to a matrix plus a separate bias vector (requiring different update rules for each). This is at the expense of needing to manipulate the inputs each time, and there is no great difference between the approaches in terms of efficiency. Some libraries take one approach, some take the other.