6
$\begingroup$

I would like to summarize (as in R) the contents of a CSV (possibly after loading it, or storing it somewhere, that's not a problem). The summary should contain the quartiles, mean, median, min and max of the data in a CSV file for each numeric (integer or real numbers) dimension. The standard deviation would be cool as well.

I would also like to generate some plots to visualize the data, for example 3 plots for the 3 pairs of variables that are more correlated (correlation coefficient) and 3 plots for the 3 pairs of variables that are least correlated.

R requires only a few lines to implement this. Are there any libraries (or tools) that would allow a similarly simple (and efficient if possible) implementation in Java or Scala?

PD: This is a specific use case for a previous (too broad) question.

$\endgroup$
  • $\begingroup$ I found a thread discussing something similar for cascading, mahout, hadoop and some other technologies. I have to check it into detail, but I can't now, so I'll simply leave it here... $\endgroup$ – Trylks Aug 31 '14 at 2:13
  • $\begingroup$ Turns out that since 4 days ago Spark supports data frames and probably loading from csv files, which is something. I will spend some time checking that. $\endgroup$ – Trylks Mar 17 '15 at 11:45
2
$\begingroup$

Checkout Breeze and apache commons math for the maths, and ScalaLab for some nice examples of how to plot things in Scala.

I've managed to get an environment setup where this would just be a couple of lines. I dont actually use ScalaLab, rather borrow some of its code, I use Intellij worksheets instead.

$\endgroup$
  • $\begingroup$ Thx a lot. I've seen Breeze can read a CSV and can calculate several statistics like mean and variance. The quartiles are missing, but this may be the best in the state of the art for Scala, a bit far from R. $\endgroup$ – Trylks Aug 30 '14 at 2:14
  • $\begingroup$ @Trylks R is a DSL for statistics, so out of box it will have a lot of helper functions for every day work for statiticians, but its totally inappropriate for building a product, doing big data or writting complicated machine laerning algorithms in a scalable way. ScaLa is a Scalable Language and is designed to be expanded to whatever people want. You can do just about anything in Scala, but it wont all be out-of-box. The payoff is massive though, once u climb a little way up the learning curve one realizes how much more powerful it is. $\endgroup$ – samthebest Aug 30 '14 at 5:42
  • $\begingroup$ I know, that's why I'm asking for Scala libraries :) Making my own library would not be wise if there were good libraries out there, which is usually the case. I found some libraries like Breeze (and Breeze-Viz, and Breeze-Bokeh), Saddle, Spire, Spark, Scalding, etc. The problem is that I'm quite uncertain about which one to choose, to use or expand, the nicest ones in philosophy seem to be dead, the most efficient ones seem to be useless, and in general I found none that would suit this simple use case :/ $\endgroup$ – Trylks Aug 31 '14 at 2:07
1
$\begingroup$

If your data is numeric, try loading it into ELKI (Java). With the NullAlgorithm it will give you scatterplots, histograms and parallel coordinate plots. It's fast in reading the data; only the current Apache Batik-based visualization is slooow because it's using SVG. :-( I'm mostly using it "headless".

It also has classes for various statistics (including higher order moments on data streams), but I havn't seen them in the default UI yet.

$\endgroup$
  • $\begingroup$ Looks cool, lots of stuff and I had no idea about that library. It seems a bit confusing, though, there is a textwriter but nothing named "reader". In particular, I don't see anything about quartiles. In this case the problem is that mixing libraries is probably not a good idea, they will have different data formats and I will need to move the data from one to another back and forth, or something even worse. $\endgroup$ – Trylks Aug 31 '14 at 1:58
  • $\begingroup$ CSV data is usually read using FileBasedDatabaseConnection and NumberVectorLabelParser. Quantiles can be determined using QuickSelect. If you don't want to mix libraries, then don't ask about libraries... $\endgroup$ – Anony-Mousse Sep 1 '14 at 8:58
  • $\begingroup$ 1. I would have never guessed those names, thank you. 2. Spark (MLib) uses Breeze over Hadoop clusters, the combination is very natural for those libraries, AFAIK, because they are quite agnostic in the information representation. When moving to higher level (aka less raw) libraries decisions will be made and that means more compatibility problems, specially when that means decisions about information representation. $\endgroup$ – Trylks Sep 1 '14 at 17:37
0
$\begingroup$

I'd have a closer look at one of Apache Spark's modules: MLlib.

$\endgroup$
  • $\begingroup$ So far, it "parses" CSV files by splitting using ",". This fails for example if there are commas inside some field delimited with quotes. Not a good start, but I will keep checking spark along other libraries. $\endgroup$ – Trylks Mar 16 '15 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.