$$\theta_t \leftarrow \theta_{t-1} - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}$$

Where $$\hat{m}_t$$ is a bias-corrected moving average of gradient means:

$$\hat{m}_t \propto \beta_1 m_{t-1} + (1 - \beta_1) g_t$$

And $$\hat{v}_t$$ is a bias-corrected moving average of uncentered variance:

$$\hat{v}_t \propto \beta_2 m_{t-1} + (1 - \beta_2) g_t^2$$

I understand the idea that, if the mean is large compared to the variance, gradients can be trusted more and you should take larger steps. But it this case, wouldn't it be more logical to use centered variance instead of uncentered variance in the above formula?

I looked for the answer and found this thread but I still wasn't convinced by the answer. Essentially the explanation was that uncentered variance is roughly equal to centered variance because the mean gradient is approximately zero. But if the mean gradient is zero, it means that you are equally likely to move in any direction, so you aren't optimizing anything...

So, why is Adam optimizer using uncentered variance? Why not centered variance which would seem a lot more logical?

• The link to the thread you're commenting on doesn't work. – Paul May 25 '19 at 15:16
• Maybe you meant this thread: forums.fast.ai/t/… – Paul May 25 '19 at 15:41