# Implementing RMSProp, but finding differences between reference versions

I am researching to implement RMSProp in a neural network project I am writing.

I have not found any published paper to refer for a canonical version - I first stumbled across the idea from a Coursera class presented by Geoffrey Hinton (lecture 6 I think). I don't think the approach has ever been formally published, despite many gradient-descent optimisation libraries having an option called "RMSProp". In addition, my searches are showing up a few variations of the original idea, and it is not clear why they differ, or whether there is a clear reason to use one version over another.

The general idea behind RMSProp is to scale learning rates by a moving average of current gradient magnitude. On each update step, the existing squared gradients are averaged into a running average (which is "decayed" by a factor) and when the network weight params are updated, the updates are divided by the square roots of these averaged squared gradients. This seems to work by stochastically "feeling out" the second order derivatives of the cost function.

Naively, I would implement this as follows:

Params:

• $\gamma$ geometric rate for averaging in [0,1]
• $\iota$ numerical stability/smoothing term to prevent divide-by-zero, usually small e.g 1e-6
• $\epsilon$ learning rate

Terms:

• $W$ network weights
• $\Delta$ gradients of weights i.e. $\frac{\partial E}{\partial W}$ for a specific mini-batch
• $R$ RMSProp matrix of running average squared weights

Initialise:

• $R \leftarrow 1$ (i.e. all matrix cells set to 1)

For each mini-batch:

• $R \leftarrow (1-\gamma)R + \gamma \Delta^2$ (element-wise square, not matrix multiply)
• $W = W - \epsilon \frac{\Delta}{\sqrt{R + \iota}}$ (all element-wise)

I have implemented and used a version similar to this before, but that time around I did something different. Instead of updating $R$ with a single $\Delta^2$ from the mini-batch (i.e. gradients summed across the mini-batch, then squared), I summed up each individual example gradient squared from the mini-batch. Reading up on this again, I'm guessing that's wrong. But it worked reasonably well, better than simple momentum. Probably not a good idea though, because of all those extra element-wise squares and sums needed, it will be less efficient if not required.

So now I am discovering further variations that seem to work. They call themselves RMSProp, and none seems to come with much rationale beyond "this works". For example, the Python climin library seems to implement what I suggest above, but then suggests a further combination with momentum with the teaser "In some cases, adding a momentum term β is beneficial", with a partial explanation about adaptable step rates - I guess I'd need to get more involved in that library before fully understanding what they are. In another example the downhill library's RMSProp implementation combines two moving averages - one is the same as above, but then another, the average of gradients without squaring is also tracked (it is squared and taken away from the average of squared weights).

I'd really like to understand more about these alternative RMSProp versions. Where have they come from, where is the theory or intuition that suggests the alternative formulations, and why do these libraries use them? Is there any evidence of better performance?

climin implements RMSProp with Nesterov momentum. Momentum methods try to avoid the oscillation that often happens with SGD by slowly changing the current direction of updates. The algorithm given in climin documentation introduces the $$\beta$$ parameter that controls how much of the previous update direction is retained. Nesterov momentum is implemented by first taking a step towards the previous update direction $$v_t$$, calculating gradient at that position, using the gradient to obtain the new update direction $$v_{t+1}$$, and finally updating the parameters. The climin implementation also includes the smoothing term $$\iota$$ inside the square root for stability (1e-8), even though it's not mentioned in the documentation.
• The variance is defined as $E[X^2]-E[X]^2$. Hinton only bothers about doing the moving average for $E[X^2]$ (second raw moment), but in that implementation they also do $E[X]^2$ (squared average), which they then subtract to obtain the variance. Dec 28, 2018 at 0:28