Understanding Learning Rate in depth

Question

I am trying to understand why the learning rate does not work universally. I have two different data sets and have tested out three learning rates 0.001 ,0.01 and 0.1 . For the first data set, I was able to achieve results for all learning rates at optimization using stochastic gradient descent.

For the second data set the learning rate 0.1 did not converge. I understand the logic behind it overshooting the gradients, however, I'm failing to understand why this was the case for one data set but not the other? I was unable to find much about this online but I have been advised that it was due to the data shape and potentially requires more deep insight into the data.

If there is any relevant literature to read that would be highly appreciated.

Answer 1

It depends on the loss function, The loss function of data set 1 might have a different shape than the loss function of a different dataset, for example, they might one might be a simple convex function while the other will be a crazy non-convex function with saddle points and stuff So a learning rate for one data set might lead to its optimum, however, the same learning rate might not be suitable for the other loss function. Try to visualize different loss functions and it will be clear.