1
$\begingroup$

Can someone explain RMSprop in layman's terms? I have tried reading various resources but they don't talk about the intuition on couple of things:

1) Why is is different from SGD with momentum? 2) Why does it actually work?

The only literature I could find were the slides of the professor who wrote this optimization technique and the slides didn't make a lot of sense to me.

Thanks

$\endgroup$

1 Answer 1

1
$\begingroup$

What momentum does is to average out the derivatives in different directions in parameter space so that the directions that oscillate much will be close to zero, but for the parameters that do not oscillate this would not be the case - averaging will not vanish.

On the other hand RMSprop works as a dampener in all directions: larger the derivatives, larger the dampening rate. This means that the fast oscillatory modes with large derivatives will be dampened more. The learning in slow directions which you want to keep may also be dampened, but the rate will be smaller.

You can find more details on RMSprop and their combination Adam on the following Coursera course:

https://www.coursera.org/learn/deep-neural-network/lecture/y0m1f/gradient-descent-with-momentum

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.