Since, there are so many local minimums in so complex neural network function, it is common for a neural network to get stuck on a local minimum. How will the neural network get unstuck from that local minimum.

enter image description here

  • 1
    $\begingroup$ We use momentum-based optimization techniques to avoid local minimums. Some other variants of momentum-based optimizers are available which solve the problem of overshooting the global minimum. Refer this. $\endgroup$ – Shubham Panchal Oct 6 '20 at 15:22
  • $\begingroup$ every once in a while introduce noise to jolt the optimizer out of its comfort zone $\endgroup$ – Scott Stensland Oct 6 '20 at 17:43

In various ways suchs as

  • momentum: think of momentum as a means of dampening oscillations and speeding up the iterations, leading to faster convergence. This means replacing gradients with a leaky average over past gradients.
  • sparse features and preconditioning (Adagrad): decrease the learning rate dynamically on a per-coordinate basis. This means, use the magnitude of the gradient as a means of adjusting how quickly progress is achieved - coordinates with large gradients are compensated with a smaller learning rate.
  • RMSProp: a combination of momentul and adagrad, combining leaky averages and coefficient-wise preconditioner
  • Adadelta: The learning rate cannot be parameterised but rather adapts itself given rate of change in the model parameters
  • Adam: a great algorithm that summarises features from all of the above
  • Scheduling: decreasing the learning rate during training

sources: https://distill.pub/2017/momentum/, https://d2l.ai/chapter_optimization


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy