1
$\begingroup$

In Neural Networks and Deep Learning, the gradient descent algorithm is described as going on the opposite direction of the gradient.

Link to place in book. What prevents this strategy from landing in a local minimum?

$\endgroup$
1
  • $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Jun 19, 2022 at 19:24

1 Answer 1

3
$\begingroup$

It does not. Gradient descent is not immune to local minima in non-convex function optimization.

Nevertheless, the noise introduced by stochastic gradient descent (SGD) helps escaping local minima. Other hyperparameters, like the learning rate, momentum, etc, also help.

You can check sections 4.1 and 4.2 of Stochastic Gradient Learning in Neural Networks for detailed explanations and mathematical formulation of SGD and its convergence properties.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.