According to questions on the internet, the bias is a learnable parameter, and there are different solutions to updating it, but I failed to find a concise methodology of correctly updating biases during training.

When I tried to overfit a small network, it failed when the bias was introduced into the training: enter image description here

When I tried to scale down bias updates it produeced similar, but delayed results: enter image description here

Next I tried to make bias updates porportional to the train set error, again only delaying the trend: enter image description here

Next I tried to make bias updated inversely porportional to the error in the training set, but I suppose this would not have any shown benefits without a validation set. Alas the effect was the same: enter image description here

According to @Noah Weber, Bias is something that would help in reducing overfitting during training, which is actually consistent with my previous experiments.

Based on this I would suppose the more overfitting occurs, the more the bias term should be updated. This can clearly be measured by the differences in the error and test set. Should bias be updated according to that?

  • $\begingroup$ Can you provide your code? $\endgroup$ – PascalIv Mar 11 '20 at 10:07
  • $\begingroup$ The bias gradient values might be wrong, I keep this possiblity open. Here is the code: github.com/davids91/rafko/blob/… Thank you for your input! $\endgroup$ – David Tóth Mar 11 '20 at 10:37

I think you are mixing up the bias of a model as in here, with the bias terms of a neural network which are just the constant term of the linear model of each layer. Updating the biases for training will not reduce overfitting since each bias is an additional parameter of the model. Remember that the weights (and the bias is also a weight) are updated proportionally to the negative gradient of the loss function. Therefore there must be an error in your implementation since the training error gets larger which is highly unlikely for gradient descent (unless your learning rate is far too high).

  • $\begingroup$ The implementation might be wrong, the bias' gradient is alculated via its neuron's back-propagated error. So for each weight update: the bias is updated with the coresponding error term of the Neuron mulitplied with the actual step size. Should that principle be correct? $\endgroup$ – David Tóth Mar 11 '20 at 10:35
  • $\begingroup$ "its neuron's back-propagated error" should be the "negative gradient of the error/ loss function w.r.t the bias weight" Just to be sure, did you maybe forget the minus? This would be consistent with your observation of an increase in the error function when the bias is updated $\endgroup$ – PascalIv Mar 12 '20 at 11:19
  • 1
    $\begingroup$ Yes, you were right, I accepted your answer. However, Gradient descent can increase slightly, and depending on the actual algorithm use, could produce noisy cost outputs. The problem was in my implementation of the algorithm. Thank you for your confirmation! $\endgroup$ – David Tóth Mar 12 '20 at 14:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.