Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Can someone explain RMSprop in layman's terms? I have tried reading various resources but they don't talk about the intuition on couple of things:
1) Why is is different from SGD with momentum? 2) Why does it actually work?
The only literature I could find were the slides of the professor who wrote this optimization technique and the slides didn't make a lot of sense to me.
Thanks
What momentum does is to average out the derivatives in different directions in parameter space so that the directions that oscillate much will be close to zero, but for the parameters that do not oscillate this would not be the case - averaging will not vanish.
On the other hand RMSprop works as a dampener in all directions: larger the derivatives, larger the dampening rate. This means that the fast oscillatory modes with large derivatives will be dampened more. The learning in slow directions which you want to keep may also be dampened, but the rate will be smaller.
You can find more details on RMSprop and their combination Adam on the following Coursera course:
https://www.coursera.org/learn/deep-neural-network/lecture/y0m1f/gradient-descent-with-momentum
