So I was going through Adaptive Gradient Descent, and learning the intuition behind it: optimizing the learning algorithm, and getting the model to converge faster. The way AdaGrad does this, is by dividing the weight vector by the $L_2$ norm. The equation (copied from here), for it, is:

$\theta_{t+1} = \theta_{t} - \dfrac{\eta}{\sqrt{G_t}} \odot g_{t}$

where, if i understand correctly, $\sqrt{G_t}$, is the root of the sum of the square of the gradients, which is the $L_2$ norm. Now if I understand correctly, $g_t$ in this case would be considered the "loss function"

Then I got introduced to Ridge Regression, which basically adds the $L_2$ norm (scaled by a lambda) to the loss function. The intuition behind this is smoothing the loss function so that it doesn't overfit the data. The equation, (reworked from here) is:

$Loss_{L_2} = Loss + \lambda L_2$

So my question, assuming all my understanding so far is correct, is: What is the intuition behind adding as opposed to dividing by the $L_2$ norm?

I understand that you can regularize the loss function in AdaGrad and that they are two separate things, but I still am having trouble wrapping my head around how the math works the way it does. Like why does dividing by the $L_2$ norm make it converge faster? Why doesn't this just smooth the weight vector as well? What is going on here intuitively?

  • $\begingroup$ Regarding your proposed alternative to ridge regression: they're equivalent, modulo a logarithmic/exponential transformation. Read about Bayesian priors for a probabilistic motivation. You will understand that it is equivalent to imposing a Gaussian prior. stats.stackexchange.com/questions/163388/… $\endgroup$
    – Emre
    Jul 13, 2018 at 17:40

2 Answers 2


Adagrad is completely unrelated to ridge regression. There is no reason to expect that there should be any similarity in their methods.

Adagrad divides by the L2 norm of the gradient. Ridge regression modifies the objective function by adding the L2 norm of the parameters. So even though it looks like the same thing, it's really not; those are two unrelated values.

There's no principle in mathematics that the L2 norm always has to be added or divided. It's just a number. It's like seeing one equation where you multiply by 2 and another equation where you add 2 and asking "why does the second one add 2 rather than multiplying by 2?" -- the answer is "why not? there's no reason to expect them to have to use the number 2 in the same way".

  • $\begingroup$ I think that the middle paragraph is what gave me that eureka moment. I thought that the two L2 norms were the same, the fact that it is the norm of the gradient smoothes out the rate of change of the weight vector so it makes training more optimized, whereas adding the norm to the cost function smoothes out the entire shape of the function by adding the norm. <-- that is the key intuition I was looking for, and you helped me get there, so this is why I'm accepting your answer. Thanks $\endgroup$
    – JoeVictor
    Jul 13, 2018 at 19:00

Sparse features have small number of updates relative to dense ones, and this is a problem.

The idea of AdaGrad is to make the update speed (learning rate) of a parameter is inversely proportional to the sum of update history of that parameter ($\eta$ / sum of previous updates).

enter image description here

This makes the small number of updates done to sparse features have a higher learning rate than dense ones.

For more details, you can watch the first 10 minutes of this video.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.