I have found a number of libraries and tools for data science in Scala, I would like to know about which one has more adoption and which one is gaining adoption at a faster pace and to what extent this is the case. Basically, which one should I bet for (if any at this point).

Some of the tools I've found are (in no particular order):

  • Scalding
  • Breeze
  • Spark
  • Saddle
  • H2O
  • Spire
  • Mahout
  • Hadoop
  • MongoDB

If I need to be more specific to make the question answerable: I'm not particularly interested in clusters and Big Data at this moment, but I'm interested in sizable data (up to 100 GB) for information integration and predictive analytics.

  • $\begingroup$ The question contains all kinds of types of technologies. It is like comparing apples to cars and asking which one has a better price-to-value ratio. $\endgroup$ – Make42 Nov 10 '16 at 9:48
  • $\begingroup$ In Python there are clear winners like Pandas and Scikit. Which libraries would be the Pandas and the Scikit of Scala? $\endgroup$ – Trylks Nov 10 '16 at 18:56
  • $\begingroup$ Now you are talking about at least only two types of libraries: numeric calculation (scikit) and table collections (pandas). Here is what the libraries you mentioned in your question are: Scalding (processing framework, functional API-layer for Hadoop), Breeze (linear algebra collections), Spark (processing framework), Saddle (table collections), H2O (a platform for ML), Spire (math collections), Mahout (ML library for Hadoop), Hadoop (batch processing framework), MongoDB (NoSQL document database). Scalding, Breeze, Sadle, Spire only are Scala-programming related. The others aren't. $\endgroup$ – Make42 Nov 14 '16 at 12:23
  • $\begingroup$ I haven't used table libraries outside of R so far, but for Scala darrenjw.wordpress.com/2015/08/21/… might give you a head start. $\endgroup$ – Make42 Nov 14 '16 at 12:27
  • $\begingroup$ When you write "scikit" - what do you mean? Do you mean "scipy" or all scikits from scikits.appspot.com/scikits or do you mean the specifi scikit "scikit-learn"? Scipy is more an ecosystem of libraries than anything else. $\endgroup$ – Make42 Nov 14 '16 at 12:30

Not sure anybody have worked with all these tools, so I'm going to share my experience with some of them and let others share their experience with the others.

MongoDB addresses problems that involve heterogeneous and nested objects, while data mining mostly works with simple tabular data. MongoDB is neither fast with this type of data, nor provide any advanced tools for analysis (correct me if you know any). So I can think of a very few applications for Mongo in data mining.

Hadoop is a large ecosystem, containing dozens of different tools. I will assume that you mean core Hadoop features - HDFS and MapReduce. HDFS provides flexible way to store large amounts of data, while MapReduce gives basis for processing them. It has its clear advantages for processing multi-terabyte datasets, but it also has significant drawbacks. In particular, because of intensive disk IO during MapReduce tasks (that slows down computations a lot) it is terrible for interactive development, iterative algorithms and working with not-that-big datasets. For more details see my earlier answer.

Many algorithms in Hadoop require multiple MapReduce jobs with complicated data flow. This is where Scalding gets shiny. Scalding (and underlying Java's Cascading) provides much simpler API, but at the moment uses same MapReduce as its runtime and thus holds all the same issues.

Spark addresses exactly these issues. It drops Hadoop's MapReduce and offers completely new computational framework based on distributed in-memory collections and delayed evaluations. Its API is somewhat similar to Scalding's with all MR complexity removed, so it's really easy to get started with it. Spark is also the first in this list that comes with data mining library - MLlib.

But Spark doesn't reinvent things like basic linear algebra. For this purpose it uses Breeze. To my opinion, Breeze is far in quality from scientific packages like SciPy, Octave or Julia, but it is still good enough for most practical use cases.

Mahout relies on Hadoop's MapReduce and thus is terrible for iterative algorithms. Spire and Saddle look cute and probably have their niche, but seem to be much less well-known than Breeze. I couldn't find much information about H2O, so it doesn't look like a big player here (comments from people who used it are welcome).

Some quick summary. Spark seems to be the most simple, flexible and fast-growing project for large-scale data processing. It facilitates a number of new projects (e.g. Shark or Spark SQL) and penetrates into existing (including Cascading and Mahout). Spark knows how to utilize HDFS API, and thus scales to terabytes of data easily. And for data mining downshifters who don't want to bother with a cluster setup there's always pure Breeze.


Not the answer you're looking for? Browse other questions tagged or ask your own question.