17
$\begingroup$

I use Libsvm to train data and predict classification on semantic analysis problem. But it has a performance issue on large-scale data, because semantic analysis concerns n-dimension problem.

Last year, Liblinear was release, and it can solve performance bottleneck. But it cost too much memory. Is MapReduce the only way to solve semantic analysis problem on big data? Or are there any other methods that can improve memory bottleneck on Liblinear?

$\endgroup$
11
$\begingroup$

Note that there is an early version of LIBLINEAR ported to Apache Spark. See mailing list comments for some early details, and the project site.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your answer. It looks like different from SVM. I'll survey it. :) $\endgroup$ – Puffin GDI May 14 '14 at 15:32
  • 4
    $\begingroup$ Just a reminder that we don't encourage linking off-site to an answer because its easy for links to break, causing an otherwise useful community resource to instead turn into a dead end. It's always best to put the answer directly into your post. $\endgroup$ – Ana May 14 '14 at 17:30
  • 1
    $\begingroup$ Agree with that. At this point, it barely exists as more than that link anyhow. I will add a link to the underlying project. $\endgroup$ – Sean Owen May 14 '14 at 21:02
10
$\begingroup$

You can check out vowpal wabbit. It is quite popular for large-scale learning and includes parallel provisions.

From their website:

VW is the essence of speed in machine learning, able to learn from terafeature datasets with ease. Via parallel learning, it can exceed the throughput of any single machine network interface when doing linear learning, a first amongst learning algorithms.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Open source and some wiki. It looks good. Thanks for your suggestion. :) $\endgroup$ – Puffin GDI May 15 '14 at 1:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.