According to questions on the internet, the bias is a learnable parameter, and there are different solutions to updating it, but I failed to find a concise methodology of correctly updating biases during training.
When I tried to overfit a small network, it failed when the bias was introduced into the training:
According to @Noah Weber, Bias is something that would help in reducing overfitting during training, which is actually consistent with my previous experiments.
Based on this I would suppose the more overfitting occurs, the more the bias term should be updated. This can clearly be measured by the differences in the error and test set. Should bias be updated according to that?