From some neural net article I read that if you scale up the Neural Net architecture the differnce in different local minimas in the loss surface diminishes. Essentially all local minimas become equivalent.

If that is the case then why do different variants of SGD (like Adagrad, ADAM etc.) works better than plain SGD? I believe reason for using these variants of SGD is to solve "bad" local minima problem, but if all local minimas are more or less the same then what is objective of using these variants?


I believe reason for using these variants of SGD is to solve "bad" local minima problem

That is not accurate. The variants are mostly about accelerating steps of gradient descent when faced with shallow or rapidly changing gradients, or with gradients that need adaptive learning rates because some parts of the network get stronger gradient signals than others. They do this by weighting or adjusting the step sizes in each dimension, based on additional knowledge to the current gradients, such as the history of previous gradients.

Shapes like saddle points or curving gullies, in the cost function "landscape" can cause difficulties for basic SGD. Take a saddle point as an example - there is no "bad" minima, a saddle point can be quite high up in a cost function. But the gradient values can be very low, even if they pick up again if you can take steps away from the saddle point. The trouble for SGD is that using just the gradient is likely to make updates oscillate up and down the steep parts of the saddle, and not move in the shallow direction away from the saddle point. Other difficult shapes can cause similar problems.

For a visualisation of the difference in behaviours of some of the optimisers at a saddle point, take a look at this animation (I'd like to find a creative-commons variant of this and include in the answer), and a blog that references it.

In addition, for deep learning network, there is a problem that gradients can "explode" or "vanish" as you work back through the layers. Using ReLU activation or similar can help with that, but is not always possible (think of RNNs which need sigmoid activations inside LSTM modules). An optimiser like RMSprop deals with this by normalising gradients used by the weight update steps based on recent history. Whilst SGD could more easily get stuck and fail to update the lower layer weights (weights closer to the input) - either not updating them much at all, or taking too large steps.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.