In Neural Networks and Deep Learning, the gradient descent algorithm is described as going on the opposite direction of the gradient.
Link to place in book. What prevents this strategy from landing in a local minimum?
1
-
$\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$– Community BotJun 19, 2022 at 19:24
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
It does not. Gradient descent is not immune to local minima in non-convex function optimization.
Nevertheless, the noise introduced by stochastic gradient descent (SGD) helps escaping local minima. Other hyperparameters, like the learning rate, momentum, etc, also help.
You can check sections 4.1 and 4.2 of Stochastic Gradient Learning in Neural Networks for detailed explanations and mathematical formulation of SGD and its convergence properties.
