I am new to deep learning and my understanding of how optimizers work might be slightly off. Also, sorry for a third-grader quality of images.

For example if we have simple task our loss to weight function might look like this: simple case

As far as I understand optimizers look for improvements and try to fall into the hole that it found.

But what if we have lots of local minimas, how do I know if, for example, adam optimizer have found global minima of loss, not just some local minima? harder case

And the third case I can think of is what if we have a flat plateau of a loss function, except for a tiny range of weights, would it be found using adam? How do I know if it even exists? hardest case

Are there any tools or methods that I can use to analyse this function?


1 Answer 1


No optimizer can guarantee you that it has found the global minimum. That's why randomly initialize weight to start at different arbitrary points and then start descending towards the minima, hoping we might go to the global minima. Sometimes our stepsize is large enough to overshoot a valley of local minima and jump across it. It depends on the optimizer and steps size. But, in most practical applications, the minima found by these optimizers ( most likely to be local minima ) work well for our applications, but they are not necessarily the global minima.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.