I noticed that some popular deep learning frameworks like Keras or Pytorch allow you to set different learning rate for each layer.
What are the benefits of that approach?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
In trivial update rules like gradient descent, the learning rate is important and it somehow specifies the speed you go downhill. In popular papers like Adam optimisation technique, and in non-paperised(!) popular solution namely RMSProp the authors cared that the slope of different features may vary differently and in a direction you may need to go faster due to its slope. Consequently, They decided to set the learning rate and update each parameter based on its own slope and this learning rate is somehow affected by the slope of each direction independently to the other dimensions. The motivation is this. As far as I know, you just need to set the learning rate for your optimisation and it will be adapted by itself.