I am used to of using learning rates 0.1 to 0.001 or something, now i was working on a siamese net work with sonar images. Was training too fast, overfitting after just 2 epochs. I tried to slow the learning rate lower and lower and I can report that the network still trains with Adam optimizer with learning rate 1e-5 and decay 1e-6. But I am curious if this is a good practice to use the learning rates so low? I am using keras


1 Answer 1


I think that for the most part, the ends justify the means when it comes to learning rates. If the network is training well and you're confident that you're evaluating its generalization properly, use what works.

With that said, overfitting isn't usually caused by high learning rate. It's hard to say with the information you've supplied here, but I suspect that your regularization isn't set up well enough, so using a high learning rate overfits the poorly regularized network rapidly, while training with a low learning rate doesn't get close to a local optimum because it can't move fast enough, but it's still heading towards an overfit optimum.

I'd check in on your regularization. What are you using?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.