0
$\begingroup$

According to questions on the internet, the bias is a learnable parameter, and there are different solutions to updating it, but I failed to find a concise methodology of correctly updating biases during training.

When I tried to overfit a small network, it failed when the bias was introduced into the training: enter image description here

When I tried to scale down bias updates it produeced similar, but delayed results: enter image description here

Next I tried to make bias updates porportional to the train set error, again only delaying the trend: enter image description here

Next I tried to make bias updated inversely porportional to the error in the training set, but I suppose this would not have any shown benefits without a validation set. Alas the effect was the same: enter image description here

According to @Noah Weber, Bias is something that would help in reducing overfitting during training, which is actually consistent with my previous experiments.

Based on this I would suppose the more overfitting occurs, the more the bias term should be updated. This can clearly be measured by the differences in the error and test set. Should bias be updated according to that?

Improve this question
$\endgroup$
2
  • $\begingroup$ Can you provide your code? $\endgroup$
    – PascalIv
    Mar 11, 2020 at 10:07
  • $\begingroup$ The bias gradient values might be wrong, I keep this possiblity open. Here is the code: github.com/davids91/rafko/blob/… Thank you for your input! $\endgroup$
    – Dávid Tóth
    Mar 11, 2020 at 10:37

1 Answer 1

Reset to default
1
$\begingroup$

I think you are mixing up the bias of a model as in here, with the bias terms of a neural network which are just the constant term of the linear model of each layer. Updating the biases for training will not reduce overfitting since each bias is an additional parameter of the model. Remember that the weights (and the bias is also a weight) are updated proportionally to the negative gradient of the loss function. Therefore there must be an error in your implementation since the training error gets larger which is highly unlikely for gradient descent (unless your learning rate is far too high).

Improve this answer
$\endgroup$
3
  • $\begingroup$ The implementation might be wrong, the bias' gradient is alculated via its neuron's back-propagated error. So for each weight update: the bias is updated with the coresponding error term of the Neuron mulitplied with the actual step size. Should that principle be correct? $\endgroup$
    – Dávid Tóth
    Mar 11, 2020 at 10:35
  • $\begingroup$ "its neuron's back-propagated error" should be the "negative gradient of the error/ loss function w.r.t the bias weight" Just to be sure, did you maybe forget the minus? This would be consistent with your observation of an increase in the error function when the bias is updated $\endgroup$
    – PascalIv
    Mar 12, 2020 at 11:19
  • 1
    $\begingroup$ Yes, you were right, I accepted your answer. However, Gradient descent can increase slightly, and depending on the actual algorithm use, could produce noisy cost outputs. The problem was in my implementation of the algorithm. Thank you for your confirmation! $\endgroup$
    – Dávid Tóth
    Mar 12, 2020 at 14:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.