0
$\begingroup$

I am creating a relatively large hobby project in Scala that needs a few ml algorithms for text classification into topics. My dataset is not huge, it is < 500,000 items with dimensionality 5 (2 dimensions are free form text).

From what I've started with on Spark, it is heavily geared toward distributed computing and other production related concerns but it has a nice ml library. Is it worth using Spark if I only plan to run my project on a local dev machine level or is there a more appropriate library out there for me in Scala? Something like scikit-learn, but in Scala.

Improve this question
$\endgroup$
2
  • $\begingroup$ Can you use other Languages (Such as Python) for this project ? $\endgroup$
    – Shamit Verma
    Mar 8, 2019 at 13:21
  • $\begingroup$ The project is also part learning exercise which is why I wanted to use Scala as I had never used it before but heard a lot about it. I know for python there are almost too many options for this sort of thing ;) $\endgroup$
    – Matias Grioni
    Mar 9, 2019 at 5:53

2 Answers 2

Reset to default
1
$\begingroup$

You're right, Spark is intended to scale in a distributed computing environment, but it is absolutely performs well locally.

When running locally Spark will still 'distribute' processing across local Executors, you have options to control the number of CPU cores the Spark job can use, and how many cores each Executor can use. You can get a lot out of using Spark locally, and Spark makes it very easy to scale up to a cluster of machines (e.g. on AWS or Google Cloud) when your single machine can't handle the tasks and it becomes necessary.

I'd also check out SMILE as an alternative to Spark MLlib in Scala.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the answer. I will continue with Spark then (since it is so popular then and has a lot of support) and keep SMILE in mind for future projects. $\endgroup$
    – Matias Grioni
    Mar 9, 2019 at 5:51
0
$\begingroup$

Since learning is one of he objectives, Spark should work for this project. There are some drawbacks (Such as code running on CPU instead of GPU) but it should work.

For text classification; High level flow could be :

  1. Build a Language Model + Vector representation of words and sentences (https://spark.apache.org/docs/2.1.1/api/java/org/apache/spark/mllib/feature/Word2Vec.html)
  2. Build a classifier with vector representation of text.

Since Spark lacks good LSTM implementation; it might not perform as well as others.

One way to validate is to compare performance of such classifier on public dataset such as Yelp review dataset.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.