In Neural Networks and Deep Learning, the gradient descent algorithm is described as going on the opposite direction of the gradient.
Link to place in book. What prevents this strategy from landing in a local minimum?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this communityIn Neural Networks and Deep Learning, the gradient descent algorithm is described as going on the opposite direction of the gradient.
Link to place in book. What prevents this strategy from landing in a local minimum?
It does not. Gradient descent is not immune to local minima in non-convex function optimization.
Nevertheless, the noise introduced by stochastic gradient descent (SGD) helps escaping local minima. Other hyperparameters, like the learning rate, momentum, etc, also help.
You can check sections 4.1 and 4.2 of Stochastic Gradient Learning in Neural Networks for detailed explanations and mathematical formulation of SGD and its convergence properties.