I use Libsvm to train data and predict classification on semantic analysis problem. But it has a performance issue on large-scale data, because semantic analysis concerns n-dimension problem.

Last year, Liblinear was release, and it can solve performance bottleneck. But it cost too much memory. Is MapReduce the only way to solve semantic analysis problem on big data? Or are there any other methods that can improve memory bottleneck on Liblinear?


2 Answers 2


Note that there is an early version of LIBLINEAR ported to Apache Spark. See mailing list comments for some early details, and the project site.

  • $\begingroup$ Thanks for your answer. It looks like different from SVM. I'll survey it. :) $\endgroup$
    – Puffin GDI
    Commented May 14, 2014 at 15:32
  • 4
    $\begingroup$ Just a reminder that we don't encourage linking off-site to an answer because its easy for links to break, causing an otherwise useful community resource to instead turn into a dead end. It's always best to put the answer directly into your post. $\endgroup$
    – Ana
    Commented May 14, 2014 at 17:30
  • 1
    $\begingroup$ Agree with that. At this point, it barely exists as more than that link anyhow. I will add a link to the underlying project. $\endgroup$
    – Sean Owen
    Commented May 14, 2014 at 21:02

You can check out vowpal wabbit. It is quite popular for large-scale learning and includes parallel provisions.

From their website:

VW is the essence of speed in machine learning, able to learn from terafeature datasets with ease. Via parallel learning, it can exceed the throughput of any single machine network interface when doing linear learning, a first amongst learning algorithms.

  • 1
    $\begingroup$ Open source and some wiki. It looks good. Thanks for your suggestion. :) $\endgroup$
    – Puffin GDI
    Commented May 15, 2014 at 1:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.