3
$\begingroup$

I noticed that some popular deep learning frameworks like Keras or Pytorch allow you to set different learning rate for each layer.

What are the benefits of that approach?

$\endgroup$
0
$\begingroup$

In trivial update rules like gradient descent, the learning rate is important and it somehow specifies the speed you go downhill. In popular papers like Adam optimisation technique, and in non-paperised(!) popular solution namely RMSProp the authors cared that the slope of different features may vary differently and in a direction you may need to go faster due to its slope. Consequently, They decided to set the learning rate and update each parameter based on its own slope and this learning rate is somehow affected by the slope of each direction independently to the other dimensions. The motivation is this. As far as I know, you just need to set the learning rate for your optimisation and it will be adapted by itself.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.