Implementing RMSProp, but finding differences between reference versions

Question

I am researching to implement RMSProp in a neural network project I am writing.

I have not found any published paper to refer for a canonical version - I first stumbled across the idea from a Coursera class presented by Geoffrey Hinton (lecture 6 I think). I don't think the approach has ever been formally published, despite many gradient-descent optimisation libraries having an option called "RMSProp". In addition, my searches are showing up a few variations of the original idea, and it is not clear why they differ, or whether there is a clear reason to use one version over another.

The general idea behind RMSProp is to scale learning rates by a moving average of current gradient magnitude. On each update step, the existing squared gradients are averaged into a running average (which is "decayed" by a factor) and when the network weight params are updated, the updates are divided by the square roots of these averaged squared gradients. This seems to work by stochastically "feeling out" the second order derivatives of the cost function.

Naively, I would implement this as follows:

Params:

• $\gamma$ geometric rate for averaging in [0,1]
• $\iota$ numerical stability/smoothing term to prevent divide-by-zero, usually small e.g 1e-6
• $\epsilon$ learning rate

Terms:

• $W$ network weights
• $\Delta$ gradients of weights i.e. $\frac{\partial E}{\partial W}$ for a specific mini-batch
• $R$ RMSProp matrix of running average squared weights

Initialise:

• $R \leftarrow 1$ (i.e. all matrix cells set to 1)

For each mini-batch:

• $R \leftarrow (1-\gamma)R + \gamma \Delta^2$ (element-wise square, not matrix multiply)
• $W = W - \epsilon \frac{\Delta}{\sqrt{R + \iota}}$ (all element-wise)

I have implemented and used a version similar to this before, but that time around I did something different. Instead of updating $R$ with a single $\Delta^2$ from the mini-batch (i.e. gradients summed across the mini-batch, then squared), I summed up each individual example gradient squared from the mini-batch. Reading up on this again, I'm guessing that's wrong. But it worked reasonably well, better than simple momentum. Probably not a good idea though, because of all those extra element-wise squares and sums needed, it will be less efficient if not required.

So now I am discovering further variations that seem to work. They call themselves RMSProp, and none seems to come with much rationale beyond "this works". For example, the Python climin library seems to implement what I suggest above, but then suggests a further combination with momentum with the teaser "In some cases, adding a momentum term β is beneficial", with a partial explanation about adaptable step rates - I guess I'd need to get more involved in that library before fully understanding what they are. In another example the downhill library's RMSProp implementation combines two moving averages - one is the same as above, but then another, the average of gradients without squaring is also tracked (it is squared and taken away from the average of squared weights).

I'd really like to understand more about these alternative RMSProp versions. Where have they come from, where is the theory or intuition that suggests the alternative formulations, and why do these libraries use them? Is there any evidence of better performance?

Answer 1

RMSProp is indeed an unpublished method, and in the lecture Geoffrey Hinton gives just the general idea behind RMSProp - to divide the gradient by a moving average of the gradient magnitude. The lecture has disappeared from YouTube but you can find the slides in the end of this PDF:

https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf

When this principle is applied to Stochastic Gradient Descent, the update rule you showed is obtained. Since Hinton did not propose an exact algorithm, this principle has been applied to different optimization methods. I agree it's confusing all these methods call themselves RMSProp.

climin implements RMSProp with Nesterov momentum. Momentum methods try to avoid the oscillation that often happens with SGD by slowly changing the current direction of updates. The algorithm given in climin documentation introduces the $\beta$ parameter that controls how much of the previous update direction is retained. Nesterov momentum is implemented by first taking a step towards the previous update direction $v_t$, calculating gradient at that position, using the gradient to obtain the new update direction $v_{t+1}$, and finally updating the parameters. The climin implementation also includes the smoothing term $\iota$ inside the square root for stability (1e-8), even though it's not mentioned in the documentation.

The implementation in the Downhill library is based on the algorithm described in the paper by A. Graves. In the article (equations 38–40) the square of the average gradient is subtracted from the average square gradient. Apparently the idea is to approximate the variance of the gradient instead of its magnitude (recall that the variance of a random variable is equal to the mean of the square minus the square of the mean).