11
$\begingroup$

In reviewing “Applied Predictive Modeling" a reviewer states:

One critique I have of statistical learning (SL) pedagogy is the absence of computation performance considerations in the evaluation of different modeling techniques. With its emphases on bootstrapping and cross-validation to tune/test models, SL is quite compute-intensive. Add to that the re-sampling that's embedded in techniques like bagging and boosting, and you have the specter of computation hell for supervised learning of large data sets. In fact, R's memory constraints impose pretty severe limits on the size of models that can be fit by top-performing methods like random forests. Though SL does a good job calibrating model performance against small data sets, it'd sure be nice to understand performance versus computational cost for larger data.

What are R's memory constraints, and do they impose severe limits on the size of models that can be fit by top-performing methods like random forests?

Improve this question
$\endgroup$
1

3 Answers 3

Reset to default
10
$\begingroup$

As Konstantin has pointed, R performs all its computation in the system's memory i.e. RAM. Hence, RAM capacity is a very important constraint for computation intensive operations in R. Overcoming this constraint, data is being stored these days in HDFS systems, where data isn't loaded onto memory and program is run instead, program goes to the data and performs the operations, thus overcoming the memory constraints. RHadoop (https://github.com/RevolutionAnalytics/RHadoop/wiki) is the connector you are looking for.

Coming to the impact on algorithms which are computation intensive, Random Forests/Decision Trees/Ensemble methods on a considerable amount of data (minimum 50,000 observations in my experience) take up a lot of memory and are considerably slow. To speed up the process, parallelization is the way to go and parallelization is inherently available in Hadoop! That's where, Hadoop is really efficient.

So, if you are going for ensemble methods which are compute intensive and are slow, you would want to try out on the HDFS system which gives a considerable performance improvement.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ +1 Thank you for taking the time to improve upon the existing answer, and given in my opinion your answer is now the better answer, I've selected your answer as the answer. Cheers! $\endgroup$
    – blunders
    Jun 11, 2014 at 17:35
  • $\begingroup$ Glad to answer! $\endgroup$
    – binga
    Jun 11, 2014 at 19:01
8
$\begingroup$

R performs all computation in-memory so you can't perform operation on a dataset that is larger than available RAM amount. However there are some libraries that allow bigdata processing using R and one of popular libraries for bigdata processing like Hadoop.

Improve this answer
$\endgroup$
3
$\begingroup$

This critique is no longer justified:

While it is true that most of the standard and most respected R libraries were restricted to in-memory computations, there is a growing number of specialized libraries to deal with data that doesn't fit into memory.
For instance, for random forests on large datasets, you have the library bigrf. More info here: http://cran.r-project.org/web/packages/bigrf/

Another area of growth is R's connectedness to big data environments like hadoop, which opens another world of possibilities.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.