# What is the intuition behind Ridge Regression and Adapting Gradient Descent algorithms?

So I was going through Adaptive Gradient Descent, and learning the intuition behind it: optimizing the learning algorithm, and getting the model to converge faster. The way AdaGrad does this, is by dividing the weight vector by the $L_2$ norm. The equation (copied from here), for it, is:

$\theta_{t+1} = \theta_{t} - \dfrac{\eta}{\sqrt{G_t}} \odot g_{t}$

where, if i understand correctly, $\sqrt{G_t}$, is the root of the sum of the square of the gradients, which is the $L_2$ norm. Now if I understand correctly, $g_t$ in this case would be considered the "loss function"

Then I got introduced to Ridge Regression, which basically adds the $L_2$ norm (scaled by a lambda) to the loss function. The intuition behind this is smoothing the loss function so that it doesn't overfit the data. The equation, (reworked from here) is:

$Loss_{L_2} = Loss + \lambda L_2$

So my question, assuming all my understanding so far is correct, is: What is the intuition behind adding as opposed to dividing by the $L_2$ norm?

I understand that you can regularize the loss function in AdaGrad and that they are two separate things, but I still am having trouble wrapping my head around how the math works the way it does. Like why does dividing by the $L_2$ norm make it converge faster? Why doesn't this just smooth the weight vector as well? What is going on here intuitively?

• Regarding your proposed alternative to ridge regression: they're equivalent, modulo a logarithmic/exponential transformation. Read about Bayesian priors for a probabilistic motivation. You will understand that it is equivalent to imposing a Gaussian prior. stats.stackexchange.com/questions/163388/… – Emre Jul 13 '18 at 17:40