Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I've noticed in a few caffe models I've been working with that the learning rate for the bias is often set to be twice that of the one for the weights.
Another user mentions that this is the case in this thread but does not elaborate on what the reason for doing so is.
Is there some justification for having different learning rates for the bias as the weights? Did I miss some paper providing a theoretical or experimental justification of this?
I can't find any literature discussing setting a fixed learning rate differently for the bias specifically than for the other weights, but if you asked the researcher responsible for that model their answer would probably be something like:
"The network wasn't training properly and I noticed the error for the bias term wasn't converging as fast as I'd like. When I tried doubling the learning rate for just that term it seemed to fix the problem. I dunno man, it just worked."
As far as I can tell, training neural networks is still more art than science and the amount of skill, patience, and care taken in training a neural network (depending on the architecture) can have as much impact on the model's performance as the network topology. Consider issues like mode collapse in GANs, vanishing gradient with ReLUs, or just getting stuck in local optima. Neural networks often need to be "babysat" during training, and consequently trial and error may contribute significantly to certain training process decisions. I'm sorry if that's not a satisfactory answer, but I have a strong suspicion that's what's going on here. If you post some specific examples, it might be easier to figure out concretely what's going on, but otherwise my guess is that this was just the result of trial and error.
Regarding using parameter-specific learning rates generally, most modern optimizers do this, including AdaGrad, RMSProp and Adam.
