1
$\begingroup$

Is it correct to say that as you add more layers and more neurons, your learning rate should then decrease?

So, generally speaking, the larger the net, the smaller the learning rate?

$\endgroup$
1
$\begingroup$

The learning rate affects how much your model weights are changed with each update. So the question is what is a good learning rate? This depends on your problem space, dataset and network parameters (ie. Activation functions, complexity, regularisation, optimizers, etc).

Too high a learning rate, your model may not reach convergence due to weights updated to “overshoot” the optimum.

Too low a learning rate, your model may also not reach convergence due to taking too many epochs, as the updates to weights are tiny.

It is best to experiment with the defaults ~0.01, and adjust accordingly based on visualizing the validation loss and training loss change over epochs. Also try out different optimizers (Adam, RMSprop, etc) as these adapt your learning rate over time. Another approach is to implement a learning rate scheduler, that decays your learning rate over time - the idea being as a model nears the global minima, you want to fine tune its descent with smaller weight updates.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.