In Adam optimizer algorithm, parameter updates are computed as follows:

$\theta_t \leftarrow \theta_{t-1} - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}$

Where $\hat{m}_t$ is a bias-corrected moving average of gradient means:

$\hat{m}_t \propto \beta_1 m_{t-1} + (1 - \beta_1) g_t$

And $\hat{v}_t$ is a bias-corrected moving average of uncentered variance:

$\hat{v}_t \propto \beta_2 m_{t-1} + (1 - \beta_2) g_t^2$

I understand the idea that, if the mean is large compared to the variance, gradients can be trusted more and you should take larger steps. But it this case, wouldn't it be more logical to use centered variance instead of uncentered variance in the above formula?

I looked for the answer and found this thread but I still wasn't convinced by the answer. Essentially the explanation was that uncentered variance is roughly equal to centered variance because the mean gradient is approximately zero. But if the mean gradient is zero, it means that you are equally likely to move in any direction, so you aren't optimizing anything...

So, why is Adam optimizer using uncentered variance? Why not centered variance which would seem a lot more logical?

  • $\begingroup$ The link to the thread you're commenting on doesn't work. $\endgroup$ – Paul May 25 '19 at 15:16
  • $\begingroup$ Maybe you meant this thread: forums.fast.ai/t/… $\endgroup$ – Paul May 25 '19 at 15:41

If anything, you don't have access to the mean. When we talk about the variance, we talk about the variance of a distribution. The distribution in this case is the total collection of gradients at each step of the optimization process, which you don't know until you're done.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.