Difference between RMSProp and Momentum?

Can someone please tell me the clear difference between the approaches of RMSProp and Gradient Descent with Momentum ? Both try to achieve the same effect . One of the blogs that I read states the difference as : "RMSProp and Momentum take contrasting approaches . While momentum accelerates our search in direction of minima, RMSProp impedes our search in direction of oscillations. "

I don't get this statement . Can someone elaborate on the difference between the two ?

Optimizers evolved with small Fix/Improvement on the previous one. So, if you will read in sequence, you will have a better understanding. In this context, RMSProp was a fix on Adagrad and it was an improvement on Momentum.

Let's see this Loss surface which is like a Valley (Imagine a River)

$$\hspace{2cm}$$
$$\hspace{5cm}$$Image source - http://d2l.ai/

Momentum -
Let's start from the red-circled point. We have a very large Gradient in X2 direction and very little in X1 and global minima is towards X1.
In Momentum, we accumulated the resultant Gradient which will obviously point more towards X2.
As a result, we will move very fast towards the other side of the river and very little towards X1. As we cross the river and start moving up, counter Gradient of X2 will start minimizing the Aggregate. Remember, it's leaky aggregation i.e. recent ones have more say. At a point, it will stop and reverse.
In the whole process, we had a little movement in X1 and a lot of oscillations in X2
This was one of the points of the Author

- Manage Gradient for each coordinate separately
- Added a scaling factor in the denominator which will act as a brake. This scaling is based on the square of past Gradients.
Now X2 will have a large brake, so it will not move so fast with momentum to cross the river. Since X1 is having a very small Gradient, its scaling will be positive(if < 1) Or almost constant(if ~ 1). So, movement in X1 will be same Or even faster.
That's why the author said: "RMSProp impedes our search in direction of oscillations"

Problem with Adagrad was that it aggregates all the past Gradients for the scaling factor which causes the brake to become larger for any case after a good number of iterations even if it has not reached Global optimum.
Let's say, if Gradient is small 0.5 then also it will start dividing by 2.5 after 10 iterations. if it is large e.g. 10, then it will start dividing by 1000 after 10 iterations. Even this large gradient will become small in the subsequent iteration.

What RMSProp changed - It made the aggregation leaky i.e. recent one will be considered more (Just like Momentum does for Gradient). With this change, the aggregation will be almost constant or at least not die as fast as in Adagrad.

$$\hspace{2cm}$$
$$\hspace{5cm}$$Image source - http://d2l.ai/

• Thanks. It clear me everything. Jun 22, 2020 at 7:35

RmsProp is a adaptive Learning Algorithm while SGD with momentum uses constant learning rate. SGD with momentum is like a ball rolling down a hill. It will take large step if the gradient direction point to the same direction from previous. But will slow down if the direction changes. But it does not change it learning rate during training. But Rmsprop is a adaptive learning algorithm. That means it adapts it learning rate using a moving average of it's gradient's square value. As the value of the moving average increases, the learning rate becomes more and more small allowing the algorithm to converge.

RMSProp:

$$g = \frac{1}{m} \sum_{1}^{m} L(\hat{y},y)$$

$$r = \delta r + (1 - \delta) g \circ g$$

$$\Delta\theta = - \frac{\epsilon}{\sqrt{r+\delta}} \circ g$$

$$\theta = \theta + \delta\theta$$

Here m is the minibatch size and r is the moving average value and g is gradient and theta is parameters.

SGD With Momentum:

$$g = \frac{1}{m}\sum_{1}^{m} L(\hat{y}, y)$$

$$v = \alpha v - \epsilon g$$

$$\theta = \theta + v$$

Here v is the velocity of Momentum.

(Adapted From Deep Learning By GoodFellow)

• Does RMSProp always use adaptive learning rate ? I was learning it from Andrew NG's Deep Learning Course and didn't seem to come across it .I'm only a beginner so please clarify . Thanks Jun 21, 2020 at 17:30
• Yes. RMSProp will always use adaptive learning rate. (Quoted from Deep Learning by Ian GoodFellow,Yoshua Bengio)
– SrJ
Jun 21, 2020 at 17:35