Adagrad adapts the learning rate $ \alpha $ all along the gradient descent process, by dividing each weight on a quantity based on the sum of the previous squared gradient up to time $ t $. Therefore, the update will be large for infrequent data samples, and small for frequent ones, which makes it a perfectly suited algorithm for sparse data.
$$ \boxed{W^{(i+1)}=W^{(i)}-\frac{\alpha}{\sqrt{G_{i}+\varepsilon}} \odot \nabla_{W} L\left(W^{(i)}\right)} $$
$ \underline{where:} $
$ G_{i} $ is diagonal matrix of size $ \mathbb{R}^{F+1,F+1} $ containing in each diagonal the sum of the squared gradient of parameter $ w_{i} $, up until time t.
$ \varepsilon $ is a parameter that we add to avoid division by zero, usually fixed to $ 10^{-8} $.
$ \odot $ is the element-wise product of matrices.
I have seen in a lot of places that without the square root operation, the algorithm performs much worse, such as https://www.ruder.io/optimizing-gradient-descent/.
So I tried to set up a small experiment with the exponent of the denominator of the learning rate as a hyperparameter. The problem setup is house price prediction, and I am performing stochastic gradient descent with AdaGrad on the California Housing Dataset. Here is what I got.
Now the square root, i.e. 0.5 performs pretty well, but there is no predictable pattern for the others. Has anyone analyzed why the square-root is important, and what advantages it has? 0.8 exponent has a better convergence as it achieves optimum value from the start itself.
