# What are R's memory constraints?

In reviewing “Applied Predictive Modeling" a reviewer states:

One critique I have of statistical learning (SL) pedagogy is the absence of computation performance considerations in the evaluation of different modeling techniques. With its emphases on bootstrapping and cross-validation to tune/test models, SL is quite compute-intensive. Add to that the re-sampling that's embedded in techniques like bagging and boosting, and you have the specter of computation hell for supervised learning of large data sets. In fact, R's memory constraints impose pretty severe limits on the size of models that can be fit by top-performing methods like random forests. Though SL does a good job calibrating model performance against small data sets, it'd sure be nice to understand performance versus computational cost for larger data.

What are R's memory constraints, and do they impose severe limits on the size of models that can be fit by top-performing methods like random forests?

As Konstantin has pointed, R performs all its computation in the system's memory i.e. RAM. Hence, RAM capacity is a very important constraint for computation intensive operations in R. Overcoming this constraint, data is being stored these days in HDFS systems, where data isn't loaded onto memory and program is run instead, program goes to the data and performs the operations, thus overcoming the memory constraints. RHadoop (https://github.com/RevolutionAnalytics/RHadoop/wiki) is the connector you are looking for.

Coming to the impact on algorithms which are computation intensive, Random Forests/Decision Trees/Ensemble methods on a considerable amount of data (minimum 50,000 observations in my experience) take up a lot of memory and are considerably slow. To speed up the process, parallelization is the way to go and parallelization is inherently available in Hadoop! That's where, Hadoop is really efficient.

So, if you are going for ensemble methods which are compute intensive and are slow, you would want to try out on the HDFS system which gives a considerable performance improvement.

• +1 Thank you for taking the time to improve upon the existing answer, and given in my opinion your answer is now the better answer, I've selected your answer as the answer. Cheers! – blunders Jun 11 '14 at 17:35
For instance, for random forests on large datasets, you have the library bigrf. More info here: http://cran.r-project.org/web/packages/bigrf/