How do I know that my weights optimizer have found the best weights?

Question

I am new to deep learning and my understanding of how optimizers work might be slightly off. Also, sorry for a third-grader quality of images.

For example if we have simple task our loss to weight function might look like this:

As far as I understand optimizers look for improvements and try to fall into the hole that it found.

But what if we have lots of local minimas, how do I know if, for example, adam optimizer have found global minima of loss, not just some local minima?

And the third case I can think of is what if we have a flat plateau of a loss function, except for a tiny range of weights, would it be found using adam? How do I know if it even exists?

Are there any tools or methods that I can use to analyse this function?

Answer 1

No optimizer can guarantee you that it has found the global minimum. That's why randomly initialize weight to start at different arbitrary points and then start descending towards the minima, hoping we might go to the global minima. Sometimes our stepsize is large enough to overshoot a valley of local minima and jump across it. It depends on the optimizer and steps size. But, in most practical applications, the minima found by these optimizers ( most likely to be local minima ) work well for our applications, but they are not necessarily the global minima.