Default value of learning rate in adam optimizer - Keras

Question

I am working on a image classification problem using Transfer Learning.

Parameters used given below:
Adam optimizer with learning rate - 0.0001

adamOpti = Adam(lr = 0.0001)
model.compile(optimizer = adamOpti, loss = "categorical_crossentropy, metrics = ["accuracy"])

For testing I used adam optimizer without explicitly specifying any parameter (default value lr = 0.001).

With the default value of learning rate the accuracy of training and validation got stuck at around 50%.
And when I use learning rate = 0.0001 in the first epoch itself I could see that the accuracy is going to 90%.

Could you please help me understand
1. why with lower value of learning rate the accuracy is increasing rapidly?
2. Also which of the above used learning rate is the better?
3. How could I make use of decaying learning rate in Keras?

Thank you

Answer 1

Learning rate is a very important hyperparameter, and often requires some experimentation. There are some good Related questions here, make sure to check those out.

1. With too large a learning rate, you might bounce around an optimum, or you might start off by sling-shooting out to a part of the parameter space where the gradients vanish. (With too small, you might take too long to converge to an optimum, or you might find a poor local optimum. These effects are lessened by adams momentum effect.)

2. That said, the best learning rate generally depends on the problem. Presumably the default was chosen pretty well for general use, but your finding isn't surprising.

3. There is a parameter decay for simple decay, or you can set up a callback for step decays.

Answer 2

There is a particular library called as ReduceLROnPlateau, that will reduce the learning rate, based on the factor value you mention. And this seems working good for all problem cases.