I'm writing my own training algorithm, but I don't know how to set the bias weight.
Have I to set bias in any layer?
Must the bias weight, be updated in every layer?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
There should be a bias weight for each virtual neuron as it controls the threshold at which the neuron responds to combined input. So if your hidden layer has 100 neurons, that is 100 bias weights for that layer. Same applies to each layer.
There are usually two different approaches taken when implementing bias. You can do one or the other:
As a separate vector of bias weights for each layer, with different (slightly cut down) logic for calculating gradients.
As an additional column in the weights matrix, with a matching column of 1's added to input data (or previous layer outputs), so that the exact same code calculates bias weight gradients and updates as for connection weights.
In both cases, you only do backpropagation calculation from neuron activation deltas to the bias weight deltas, you don't need to calculate the "activation" delta for bias, because it is not something that can change, it is always 1.0. Also the bias does not contribute deltas back further to anything else.
Actually you don’t need a bias if you have back propagation with at least 1 hidden layer. For example, if your input is zero, your forward propagation will result in 0.5 (for sigmoid) but your back propagation will adjust its weight where you finally get the right answer.