I am reading a book (TensorFlow For Dummies, Matthew Scarpino), and here it says:

Adagrad methods compute subgradients instead of gradients. A subgradient is a generalization of a gradient that applies to nondifferentiable functions. This means AdaGrad methods can optimize both differentiable and nondifferentiable functions.

But I haven’t seen such a claim anywhere else. So I want to know:

Can we really optimize non-differentiable loss functions just because we are using Adagrad?


1 Answer 1


The book has a misunderstanding (but it's understandable where it came from).

If you can compute subgradients, you can use gradient descent. You don't have to use AdaGrad -- you can use any gradient method you like. Basically, you just use a subgradient in place of the gradient in the update step. See, e.g., https://en.wikipedia.org/wiki/Subgradient_method.

AdaGrad is an adjustment to gradient descent that adjusts the update step. I believe AdaGrad is orthogonal to whether you use subgradients or gradients in the update step.

I can see where the misunderstanding comes from. The original paper on AdaGrad talks about subgradient methods. However, if you read the introduction carefully, you will recognize what's going on there. Subgradient methods are a broader category, that includes both ordinary gradient descent and subgradient descent. So, the paper is just trying to be as general as possible. Their method applies both to ordinary gradient descent and also to subgradient descent.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.