RMSProp vs Momentum in Deep Learning

Question

Can someone please tell me the clear difference between the approaches of RMSProp and Gradient Descent with Momentum? Both try to achieve the same effect. One of the blogs that I read states the difference as "RMSProp and Momentum take contrasting approaches. While momentum accelerates our search in direction of minima, RMSProp impedes our search in direction of oscillations."

I don't get this statement. Can someone elaborate on the difference between the two?

Answer 1

Optimizers evolved with small Fix/Improvement on the previous one. So, if you will read in sequence, you will have a better understanding. In this context, RMSProp was a fix on Adagrad and it was an improvement on Momentum.

Let's see this Loss surface which is like a Valley (Imagine a River)

$\hspace{2cm}$
$\hspace{5cm}$Image source - http://d2l.ai/

Momentum -
Let's start from the red-circled point. We have a very large Gradient in X2 direction and very little in X1 and global minima is towards X1.
In Momentum, we accumulated the resultant Gradient which will obviously point more towards X2.
As a result, we will move very fast towards the other side of the river and very little towards X1. As we cross the river and start moving up, counter Gradient of X2 will start minimizing the Aggregate. Remember, it's leaky aggregation i.e. recent ones have more say. At a point, it will stop and reverse.
In the whole process, we had a little movement in X1 and a lot of oscillations in X2
This was one of the points of the Author

What AdaGrad did -
- Manage Gradient for each coordinate separately
- Added a scaling factor in the denominator which will act as a brake. This scaling is based on the square of past Gradients.
Now X2 will have a large brake, so it will not move so fast with momentum to cross the river. Since X1 is having a very small Gradient, its scaling will be positive(if < 1) Or almost constant(if ~ 1). So, movement in X1 will be same Or even faster.
That's why the author said: "RMSProp impedes our search in direction of oscillations"

Problem with Adagrad was that it aggregates all the past Gradients for the scaling factor which causes the brake to become larger for any case after a good number of iterations even if it has not reached Global optimum.
Let's say, if Gradient is small 0.5 then also it will start dividing by 2.5 after 10 iterations. if it is large e.g. 10, then it will start dividing by 1000 after 10 iterations. Even this large gradient will become small in the subsequent iteration.

What RMSProp changed - It made the aggregation leaky i.e. recent one will be considered more (Just like Momentum does for Gradient). With this change, the aggregation will be almost constant or at least not die as fast as in Adagrad.

$\hspace{2cm}$
$\hspace{5cm}$Image source - http://d2l.ai/

Answer 2

RmsProp is a adaptive Learning Algorithm while SGD with momentum uses constant learning rate. SGD with momentum is like a ball rolling down a hill. It will take large step if the gradient direction point to the same direction from previous. But will slow down if the direction changes. But it does not change it learning rate during training. But Rmsprop is a adaptive learning algorithm. That means it adapts it learning rate using a moving average of it's gradient's square value. As the value of the moving average increases, the learning rate becomes more and more small allowing the algorithm to converge.

RMSProp:

$ g = \frac{1}{m} \sum_{1}^{m} L(\hat{y},y) $

$ r = \delta r + (1 - \delta) g \circ g $

$ \Delta\theta = - \frac{\epsilon}{\sqrt{r+\delta}} \circ g$

$ \theta = \theta + \delta\theta $

Here m is the minibatch size and r is the moving average value and g is gradient and theta is parameters.

SGD With Momentum:

$ g = \frac{1}{m}\sum_{1}^{m} L(\hat{y}, y) $

$ v = \alpha v - \epsilon g$

$ \theta = \theta + v $

Here v is the velocity of Momentum.

(Adapted From Deep Learning By GoodFellow)

Answer 3

Momentum(1964) (Add-on):

• is the add-on optimizer to other optimizers to accelerate(speed up) convergence by mitigating fluctuation, considering the past and current gradients, giving more importance to newer gradients with EWA:

  *Memos:

  • EWA(Exponentially Weighted Average) is the algorithm to smooth a trend(to mitigate the fluctuation of a trend), considering the past and the current values, giving more importance to newer values.
  • EWA is also called EWMA(Exponentially Weighted Moving Average).

• is added to SGD() and Adam() in PyTorch.

RMSProp(2012):

• is the optimizer which can do gradient descent by automatically adapting learning rate to parameters, considering the past and current gradients, giving much more importance to newer gradients than Momentum(1964) with EWA to accelerate convergence by mitigating fluctuation. *The learning rate is not fixed.
• 's learning rate decreases as closing to a global minimum to find the optimal solution precisely.
• 's EWA is a little bit different from Momentum(1964)'s.
• is the improved version of AdaGrad(2011) which can do gradient descent by adapting learning rate to parameters, considering the past and current gradients to accelerate convergence by mitigating fluctuation. *The learning rate is not fixed.
• is RMSprop() in PyTorch.