I am new to deep learning and my understanding of how optimizers work might be slightly off. Also, sorry for a third-grader quality of images.
For example if we have simple task our loss to weight function might look like this:
As far as I understand optimizers look for improvements and try to fall into the hole that it found.
But what if we have lots of local minimas, how do I know if, for example, adam optimizer have found global minima of loss, not just some local minima?
And the third case I can think of is what if we have a flat plateau of a loss function, except for a tiny range of weights, would it be found using adam? How do I know if it even exists?
Are there any tools or methods that I can use to analyse this function?