Is it correct to say that as you add more layers and more neurons, your learning rate should then decrease?
So, generally speaking, the larger the net, the smaller the learning rate?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
The learning rate affects how much your model weights are changed with each update. So the question is what is a good learning rate? This depends on your problem space, dataset and network parameters (ie. Activation functions, complexity, regularisation, optimizers, etc).
Too high a learning rate, your model may not reach convergence due to weights updated to “overshoot” the optimum.
Too low a learning rate, your model may also not reach convergence due to taking too many epochs, as the updates to weights are tiny.
It is best to experiment with the defaults ~0.01, and adjust accordingly based on visualizing the validation loss and training loss change over epochs. Also try out different optimizers (Adam, RMSprop, etc) as these adapt your learning rate over time. Another approach is to implement a learning rate scheduler, that decays your learning rate over time - the idea being as a model nears the global minima, you want to fine tune its descent with smaller weight updates.