In theory and what people are doing (e.g. Paper) decreasing the learning rate should help the optimizer to go "deeper into the valley" and thus decrease the loss and increase the metric.
Thus, my plan was to train a neural network with a learning rate of 1 until the loss and my metric stay approx. the same for some epochs, then with 0.1, then 0.01 and so on.
However, what I'm observing is, that the loss of the model stagnates after 1000 epochs with learning rate 1 and then decreasing the learning rate multiple times doesn't improve anything.
Do I have to decrease the lr earlier, e.g. after 400 epochs when the neural network is still learning and the loss is going down? Or do you have any ideas why this happens?