When does it make sense to choose gradient descent for SVM over liblinear?

Question

I understand using gradient descent methods with SVM is intractable if you've used the kernel trick. In that case, best to use libsvm as your solver.

But in the case that you are not using a kernel and simply treating it as a linear separation problem, when does it make sense to use gradient descent as your solver?

As I see it, liblinear is $O(N)$ time and doesn't require hyperparameter tuning.

In some past tests, liblinear has converged to a lower error rate at much faster rate than gradient-based methods.

Yet, Sklearn's own tests show it can be faster in many cases.

When is it optimal to use gradient-based methods with the SVM? Is it with a certain sized dataset or data that is highly linear and convex? What heuristics or explanations are available?

Answer 1

As mentioned in the comment, if the question is when does it make sense to use coordinate descent over stochastic gradient descent, then, one advantage with coordinate descent is that, it updates only one parameter at a time. Thus, when the data has a very very large number of features, it might make sense to use Coordinate Descent over stochastic gradient descent as SGD will try to update all model parameters at the same time. Also, for huge amounts of data CD might have lower computational complexity as against SGD. Also, it makes sense to use Coordinate descent for optimization when the function is not differentiable. Although, there are ways to convert non-differentiable functions to differentiable ones, still Coordinate descent might be one of the ways to move forward.For example, coordinate descent is a good technique to optimize a loss with L1 penalty. It is considered state of the art for Lasso regression as well as for linear SVMs.