Adagrad adapts the learning rate $$\alpha$$ all along the gradient descent process, by dividing each weight on a quantity based on the sum of the previous squared gradient up to time $$t$$. Therefore, the update will be large for infrequent data samples, and small for frequent ones, which makes it a perfectly suited algorithm for sparse data.

$$\boxed{W^{(i+1)}=W^{(i)}-\frac{\alpha}{\sqrt{G_{i}+\varepsilon}} \odot \nabla_{W} L\left(W^{(i)}\right)}$$

$$\underline{where:}$$

• $$G_{i}$$ is diagonal matrix of size $$\mathbb{R}^{F+1,F+1}$$ containing in each diagonal the sum of the squared gradient of parameter $$w_{i}$$, up until time t.

• $$\varepsilon$$ is a parameter that we add to avoid division by zero, usually fixed to $$10^{-8}$$.

• $$\odot$$ is the element-wise product of matrices.

I have seen in a lot of places that without the square root operation, the algorithm performs much worse, such as https://www.ruder.io/optimizing-gradient-descent/.

So I tried to set up a small experiment with the exponent of the denominator of the learning rate as a hyperparameter. The problem setup is house price prediction, and I am performing stochastic gradient descent with AdaGrad on the California Housing Dataset. Here is what I got.

Now the square root, i.e. 0.5 performs pretty well, but there is no predictable pattern for the others. Has anyone analyzed why the square-root is important, and what advantages it has? 0.8 exponent has a better convergence as it achieves optimum value from the start itself.