# Random Forests with Big Data - number of trees v. number of observations

I frequently use Random Forest, Regularized Random Forest, Guided Random Forest, and similar tree models.

The size of the data that I'm dealing with has grown beyond what I can work around using HPC and parallelism. It's typically large due to row length (observations) not columns (features). The data is also often not normally distributed.

I have to make a choice between:

1. Running a small number of trees (i.e. 50 or less) with either complete data or a relatively large and comparative sample
2. Running several times the number of trees, but with a correspondingly scaled down sample size

There are work-arounds and for any 1 case -- for instance, I can do some ad hoc tests to see which I think will work better, but what I'm wondering is if there is a good theoretical (or robust empirical) reasoning to either guide the choice of approach over the other or to describe the tradeoff being made?

In other words, I'm hoping that someone more comfortable with the math, statistics, and theory underlying this (type of) algorithm can offer some generalizable insight.

• Mahout has a map reduce implementation. mahout.apache.org/users/classification/…. I don't know what implementation you are using, but in theory each tree can be built independently so the full data set should never need to be in memory. you could also try sub-sampling to remove some of the easily classifiable cases to even out your distribution.
– user13684
Nov 2 '15 at 21:25
• @init-random That is good to know and I will have to check that out! Still, I'm looking for more of a theoretical / statistical answer on this one, so that I'll know what's the best approach in general (or the tradeoffs of either choice) when HPC has been exhausted or when you may not have time to go that route. Nov 3 '15 at 1:12
• I can't say for certain, but I would be surprised to find the comparison you are looking for in the literature. You are basically comparing different data sets between (1) and (2) and different models. It would be best keep one stationary and then compare the differences in the moving part. I would think (2) would be your best alternative. See here www-bcf.usc.edu/~gareth/ISL/ISLR%20Fourth%20Printing.pdf p. 320. You could think of (2) as a test/train split for cross validation and choose the model which generalizes best. I would certainly test with OOB error or cross validation.
– user13684
Nov 3 '15 at 2:02
• in the vignette of randomunitforest chapter 5.3 is a piece about incremental learning. It is a useful reading, but only the start. There is little to be found at the moment. Nov 3 '15 at 20:26
•  Random Forest, Regularized Random Forest, Guided Random Forest which implementations do you think about ? Python, WEKA or R ?
– Qbik
Apr 27 '16 at 13:57

This is because changing the value of mtry, the randomly selected number of features for a new tree, is the only meaningful hyper-parameter that should impact accuracy of the model. Since averaging converges as the no. of trees increases, the no. of trees could be reduced to a point where its performance is not impacted as much. Hence, you need to iterate and choose a a limit beyond which very small number of trees may not produce a strong enough ensemble. A random forest needs works best by using more base learners for reducing the variance by averaging each individual tree's output.