According to questions on the internet, the bias is a learnable parameter, and there are different solutions to updating it, but I failed to find a concise methodology of correctly updating biases during training.
When I tried to overfit a small network, it failed when the bias was introduced into the training:
When I tried to scale down bias updates it produeced similar, but delayed results:
Next I tried to make bias updates porportional to the train set error, again only delaying the trend:
Next I tried to make bias updated inversely porportional to the error in the training set, but I suppose this would not have any shown benefits without a validation set. Alas the effect was the same:
According to @Noah Weber, Bias is something that would help in reducing overfitting during training, which is actually consistent with my previous experiments.
Based on this I would suppose the more overfitting occurs, the more the bias term should be updated. This can clearly be measured by the differences in the error and test set. Should bias be updated according to that?