Since, there are so many local minimums in so complex neural network function, it is common for a neural network to get stuck on a local minimum. How will the neural network get unstuck from that local minimum.
In various ways suchs as
- momentum: think of momentum as a means of dampening oscillations and speeding up the iterations, leading to faster convergence. This means replacing gradients with a leaky average over past gradients.
- sparse features and preconditioning (Adagrad): decrease the learning rate dynamically on a per-coordinate basis. This means, use the magnitude of the gradient as a means of adjusting how quickly progress is achieved - coordinates with large gradients are compensated with a smaller learning rate.
- RMSProp: a combination of momentul and adagrad, combining leaky averages and coefficient-wise preconditioner
- Adadelta: The learning rate cannot be parameterised but rather adapts itself given rate of change in the model parameters
- Adam: a great algorithm that summarises features from all of the above
- Scheduling: decreasing the learning rate during training