Adagrad adapts the learning rate $ \alpha $ all along the gradient descent process, by dividing each weight on a quantity based on the sum of the previous squared gradient up to time $ t $. Therefore, the update will be large for infrequent data samples, and small for frequent ones, which makes it a perfectly suited algorithm for sparse data.

$$ \boxed{W^{(i+1)}=W^{(i)}-\frac{\alpha}{\sqrt{G_{i}+\varepsilon}} \odot \nabla_{W} L\left(W^{(i)}\right)} $$

$ \underline{where:} $

  • $ G_{i} $ is diagonal matrix of size $ \mathbb{R}^{F+1,F+1} $ containing in each diagonal the sum of the squared gradient of parameter $ w_{i} $, up until time t.

  • $ \varepsilon $ is a parameter that we add to avoid division by zero, usually fixed to $ 10^{-8} $.

  • $ \odot $ is the element-wise product of matrices.

I have seen in a lot of places that without the square root operation, the algorithm performs much worse, such as https://www.ruder.io/optimizing-gradient-descent/.

So I tried to set up a small experiment with the exponent of the denominator of the learning rate as a hyperparameter. The problem setup is house price prediction, and I am performing stochastic gradient descent with AdaGrad on the California Housing Dataset. Here is what I got. plot

Now the square root, i.e. 0.5 performs pretty well, but there is no predictable pattern for the others. Has anyone analyzed why the square-root is important, and what advantages it has? 0.8 exponent has a better convergence as it achieves optimum value from the start itself.


1 Answer 1


The 0.8 exponent fits well on your dataset, but it may not fit as well as here on other datasets. Instead, 0.5 is a standard value that ensures an average good convergence.

  • $\begingroup$ Yeah that's what my question is. Is the 0.5 value purely empirical or have there been theoretical convergence analysis done on it? $\endgroup$ Mar 13 at 18:12
  • $\begingroup$ From the paper that introduces the adagrad algorithm (link) it seems that 0.5 is purely empirical. However, I read it pretty fast, so maybe I could have missed some passage that shows the convergence analysis $\endgroup$
    – Iya Lee
    Mar 13 at 18:17
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Mar 21 at 19:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.