6
$\begingroup$

I would like to summarize (as in R) the contents of a CSV (possibly after loading it, or storing it somewhere, that's not a problem). The summary should contain the quartiles, mean, median, min and max of the data in a CSV file for each numeric (integer or real numbers) dimension. The standard deviation would be cool as well.

I would also like to generate some plots to visualize the data, for example 3 plots for the 3 pairs of variables that are more correlated (correlation coefficient) and 3 plots for the 3 pairs of variables that are least correlated.

R requires only a few lines to implement this. Are there any libraries (or tools) that would allow a similarly simple (and efficient if possible) implementation in Java or Scala?

PD: This is a specific use case for a previous (too broad) question.

Improve this question
$\endgroup$
2
  • $\begingroup$ I found a thread discussing something similar for cascading, mahout, hadoop and some other technologies. I have to check it into detail, but I can't now, so I'll simply leave it here... $\endgroup$
    – Trylks
    Aug 31, 2014 at 2:13
  • $\begingroup$ Turns out that since 4 days ago Spark supports data frames and probably loading from csv files, which is something. I will spend some time checking that. $\endgroup$
    – Trylks
    Mar 17, 2015 at 11:45

3 Answers 3

Reset to default
2
$\begingroup$

Checkout Breeze and apache commons math for the maths, and ScalaLab for some nice examples of how to plot things in Scala.

I've managed to get an environment setup where this would just be a couple of lines. I dont actually use ScalaLab, rather borrow some of its code, I use Intellij worksheets instead.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thx a lot. I've seen Breeze can read a CSV and can calculate several statistics like mean and variance. The quartiles are missing, but this may be the best in the state of the art for Scala, a bit far from R. $\endgroup$
    – Trylks
    Aug 30, 2014 at 2:14
  • $\begingroup$ @Trylks R is a DSL for statistics, so out of box it will have a lot of helper functions for every day work for statiticians, but its totally inappropriate for building a product, doing big data or writting complicated machine laerning algorithms in a scalable way. ScaLa is a Scalable Language and is designed to be expanded to whatever people want. You can do just about anything in Scala, but it wont all be out-of-box. The payoff is massive though, once u climb a little way up the learning curve one realizes how much more powerful it is. $\endgroup$
    – samthebest
    Aug 30, 2014 at 5:42
  • $\begingroup$ I know, that's why I'm asking for Scala libraries :) Making my own library would not be wise if there were good libraries out there, which is usually the case. I found some libraries like Breeze (and Breeze-Viz, and Breeze-Bokeh), Saddle, Spire, Spark, Scalding, etc. The problem is that I'm quite uncertain about which one to choose, to use or expand, the nicest ones in philosophy seem to be dead, the most efficient ones seem to be useless, and in general I found none that would suit this simple use case :/ $\endgroup$
    – Trylks
    Aug 31, 2014 at 2:07
1
$\begingroup$

If your data is numeric, try loading it into ELKI (Java). With the NullAlgorithm it will give you scatterplots, histograms and parallel coordinate plots. It's fast in reading the data; only the current Apache Batik-based visualization is slooow because it's using SVG. :-( I'm mostly using it "headless".

It also has classes for various statistics (including higher order moments on data streams), but I havn't seen them in the default UI yet.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Looks cool, lots of stuff and I had no idea about that library. It seems a bit confusing, though, there is a textwriter but nothing named "reader". In particular, I don't see anything about quartiles. In this case the problem is that mixing libraries is probably not a good idea, they will have different data formats and I will need to move the data from one to another back and forth, or something even worse. $\endgroup$
    – Trylks
    Aug 31, 2014 at 1:58
  • $\begingroup$ CSV data is usually read using FileBasedDatabaseConnection and NumberVectorLabelParser. Quantiles can be determined using QuickSelect. If you don't want to mix libraries, then don't ask about libraries... $\endgroup$
    – Has QUIT--Anony-Mousse
    Sep 1, 2014 at 8:58
  • $\begingroup$ 1. I would have never guessed those names, thank you. 2. Spark (MLib) uses Breeze over Hadoop clusters, the combination is very natural for those libraries, AFAIK, because they are quite agnostic in the information representation. When moving to higher level (aka less raw) libraries decisions will be made and that means more compatibility problems, specially when that means decisions about information representation. $\endgroup$
    – Trylks
    Sep 1, 2014 at 17:37
0
$\begingroup$

I'd have a closer look at one of Apache Spark's modules: MLlib.

Improve this answer
$\endgroup$
1
  • $\begingroup$ So far, it "parses" CSV files by splitting using ",". This fails for example if there are commas inside some field delimited with quotes. Not a good start, but I will keep checking spark along other libraries. $\endgroup$
    – Trylks
    Mar 16, 2015 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.