Why is RMSProp in many cases converging faster than Momentum?


$$v_{dW} := \beta v_{dw} +(1-\beta)dW$$ $$W := W-\alpha v_{dw}$$


$$ S_{dw} := B \cdot S_{dw} + (1-B)\cdot (dW)^2$$ $$W := W- \alpha \frac{dW}{\sqrt{S_{dw}}}$$

Where $\alpha$ is the learning rate (0.01 etc), $\beta$ is the momentum term (0.9 etc), similar to B

From my point of view, both momentum and RMSProp have "tendency to keep moving". Well, I can see how RMSprop will naturally accelerate on flat surfaces due to


when $S_{dw}$ is small, but is there another benefit that RMSprop provides?


The basic intuition is that you should not have the same learning rate for different dimensions. For instance, you can have a high slope in one direction but not for another. Consequently, you should not have the same speed for the two directions. Momentum adds acceleration. Suppose gradient is your instant velocity and the average is your average velocity. Momentum is actually viscosity or somehow friction. Suppose that you are near your optimal points, your gradients become zero and you have low average which means your speed changes slowly. They have both alpha term but what is going to be used is the running average, just a kind of average which is simple to be calculated. Take a look at here and here for making an analogy.

| improve this answer | |
  • $\begingroup$ I don't think this is a proper analogy. The gradient is already an instantaneous rate of change, i.e. acceleration. Momentum adds velocity and mass (beta) so that acceleration can accumulate. $\endgroup$ – Austin Apr 3 at 1:53
  • $\begingroup$ @Austin basically the first-order derivative, gradient here, corresponds to velocity and second-order derivative, Hessian matrix, corresponds to acceleration. Did you see the links provided above? $\endgroup$ – Media Apr 3 at 4:00
  • $\begingroup$ Your analogy is backwards. Ng literally says, "... then these derivative terms you can think of as providing acceleration...", and "and these momentum terms you can think of as representing velocity". I think you're confusing the idea that momentum ADDs acceleration with the idea that momentum IS acceleration. Momentum is mass*velocity. Vanilla SGD uses ONLY acceleration, but it has no velocity, and therefore no momentum. $\endgroup$ – Austin Apr 4 at 15:50
  • $\begingroup$ and with momentum accumulation, the terminal velocity is proportional to 1/(1-beta). $\endgroup$ – Austin Apr 4 at 15:58
  • $\begingroup$ @Austin this file may help you. $\endgroup$ – Media Apr 4 at 17:01

Momentum is linear and provides speed to the update

RMSprop contributes the exponentially decaying average of past "squared gradients"

In RMS Prop By using the average, we actually try to diminish the vertical movement because they sum up to 0(approximately) while averaging.

RMS provides average to the update

Adam uses RMS prop and Momentum Speed and Average of update combined together, On an average it will speed up the direction in which more update is needed

All three are faster than Stochastic Gradient Decent without Exponential Weighted Average, Worst Case use Momentum, Dont go for normal weight updates

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.