Is There any Open Source Implementation of Large-Scale SVM?

Question

Question:

Is anyone aware of a publicly accessible python package for large scale SVMs? Thanks!

---

Why the Question Ought to be Answerable:

As has been noted (e.g., here), the SVM problem can be computationally intensive with a large dataset. However, researches have offered plenty of work-arounds. For example,

• The Wikipedia page on SVM's links the paper A Dual Coordinate Descent Method for Large-scale Linear SVM which looks like it should be very effective.
• The paper Using the Nyström Method to Speed Up Kernel Machines offers a way to compute the kernel matrix in less than cubic time (though, by my estimate, the cost of their method is still quadratic, not linear in $n$ as the paper claims...)
• A simple Google search for "large scale SVM" returns the more recent paper Recipe for Fast Large-scale SVM Training: Polishing, Parallelism, and more RAM! along with many others (another good Google query is "large scale quadratic program").

As these papers allude to, SVM's can certainly be made to work on large data sets. In my opinion, saying otherwise would be like saying "neural networks don't work on large datasets because full batch gradient descent isn't computationally feasible." Rather, plenty of work-arounds have been proposed for both large scale NN's and large scale SVM's.

It appears that open-source ML libraries implement many such work-arounds for NN's (e.g., you can do mini-batch SGD using pytorch), but lack such implementations for SVM's (e.g., I'm not aware of any pytorch implementation of the coordinate descent algorithm described in the 2008 paper linked above). This really feels to me like a "missing feature" in the major machine learning libraries. Hence my question. Thanks, again!

Answer 1

The sklearn library has a gradient descent implementation of a support vector classifier and regressor.

The classifier is SGDClassifier(loss="hinge", ...), and the regressor is SGDRegressor(loss="epsilon_insensitive", ...)).

There are other options for loss= that are SVM-related. You can also choose a loss= that yields a different model, like logistic regression. The commonality is that all configurations use gradient descent.

LinearSVC() and LinearSVR are other options available in sklearn for linear SVMs that scale efficiently.

sklearn also has kernel approximations that you can prepend onto these models to get some of the nonlinear behaviour imparted by kernel functions.

This answer has a few more details about speeding things up. The sklearn page also has usage tips.

For large and complex datasets I think there's more of a tendency to use other models, like random forests.