2
$\begingroup$

I read on ADAM optimizer, and I saw multiple quotes which say that ADAM is a combination of Momentum and RMSprop optimizers.

So if we:

  1. Set $\beta_1 = 0$ does it means that ADAM behaves exactly as RMSprop optimizer?
  2. Set $\beta_2 = 0$ does it means that ADAM behaves exactly as Momentum optimizer?
$\endgroup$

2 Answers 2

2
$\begingroup$

ADAM is an optimization algorithm that is often used in deep learning. It is a stochastic gradient descent algorithm that is based on adaptive learning rates. It combines the ideas of momentum and RMSprop, which are both methods that help accelerate the convergence of stochastic gradient descent.

Momentum is a method that helps accelerate the convergence of SGD by adding a fraction of the previous update to the current update. This fraction is called the momentum coefficient, and it is typically set to a value between 0 and 1. Momentum helps the optimizer "roll over" any local minima and helps it converge to the global minimum faster.

RMSprop, on the other hand, is a method that helps improve the convergence of SGD by using a moving average of the squared gradients to scale the learning rate. This helps the optimizer avoid oscillating or getting stuck in local minima.

ADAM combines these two methods by using exponential moving averages of the gradients and the squared gradients to scale the learning rate. This allows ADAM to adapt the learning rates of each parameter individually, which can help the optimizer converge faster.

To answer your question, if you set $\beta_{1}=0$ in ADAM, it will stop using the momentum term and will only use the RMSprop term. This means that it will behave exactly like the RMSprop optimizer. Similarly, if you set $\beta_{2}=0$ , it will stop using the RMSprop term and will only use the momentum term. This means that it will behave exactly like the Momentum optimizer.

$\endgroup$
0
$\begingroup$

I think that the current accepted answer (Dec 2022) isn't true.

If you set $\beta_2=0$, Adam doesn't behave like Momentum, it does a different thing. Say the gradient moving average at a particular step is a vector, $v_i$, and the gradient at that step is another vector $g_i$. Then Adam with $\beta_2=0$ with learning rate $\alpha$ will update the parameters $\theta_i,t$ like:

$\theta_{i,t+1} := \theta_{i,t}-\alpha\cdot\frac{v_i}{\text{abs}(g_i)}$

which is different from momentum:

$\theta_{i,t+1} := \theta_{i,t}-\alpha\cdot v_i$.

Eyeballing PyTorch's RMSProp pseudocode, https://pytorch.org/docs/stable/generated/torch.optim.RMSprop.html it looks like RMSProp without momentum and 'centered' looks like Adam with $\beta_1=0$, but that it behaves differently if you change these parameters.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.