What is the intuition behind Ridge Regression and Adapting Gradient Descent algorithms?

Question

So I was going through Adaptive Gradient Descent, and learning the intuition behind it: optimizing the learning algorithm, and getting the model to converge faster. The way AdaGrad does this, is by dividing the weight vector by the $L_2$ norm. The equation (copied from here), for it, is:

$\theta_{t+1} = \theta_{t} - \dfrac{\eta}{\sqrt{G_t}} \odot g_{t}$

where, if i understand correctly, $\sqrt{G_t}$, is the root of the sum of the square of the gradients, which is the $L_2$ norm. Now if I understand correctly, $g_t$ in this case would be considered the "loss function"

Then I got introduced to Ridge Regression, which basically adds the $L_2$ norm (scaled by a lambda) to the loss function. The intuition behind this is smoothing the loss function so that it doesn't overfit the data. The equation, (reworked from here) is:

$Loss_{L_2} = Loss + \lambda L_2$

So my question, assuming all my understanding so far is correct, is: What is the intuition behind adding as opposed to dividing by the $L_2$ norm?

I understand that you can regularize the loss function in AdaGrad and that they are two separate things, but I still am having trouble wrapping my head around how the math works the way it does. Like why does dividing by the $L_2$ norm make it converge faster? Why doesn't this just smooth the weight vector as well? What is going on here intuitively?

Answer 1

Adagrad is completely unrelated to ridge regression. There is no reason to expect that there should be any similarity in their methods.

Adagrad divides by the L2 norm of the gradient. Ridge regression modifies the objective function by adding the L2 norm of the parameters. So even though it looks like the same thing, it's really not; those are two unrelated values.

There's no principle in mathematics that the L2 norm always has to be added or divided. It's just a number. It's like seeing one equation where you multiply by 2 and another equation where you add 2 and asking "why does the second one add 2 rather than multiplying by 2?" -- the answer is "why not? there's no reason to expect them to have to use the number 2 in the same way".

Answer 2

Sparse features have small number of updates relative to dense ones, and this is a problem.

The idea of AdaGrad is to make the update speed (learning rate) of a parameter is inversely proportional to the sum of update history of that parameter ($\eta$ / sum of previous updates).

This makes the small number of updates done to sparse features have a higher learning rate than dense ones.

For more details, you can watch the first 10 minutes of this video.