6
$\begingroup$

I frequently use Random Forest, Regularized Random Forest, Guided Random Forest, and similar tree models.

The size of the data that I'm dealing with has grown beyond what I can work around using HPC and parallelism. It's typically large due to row length (observations) not columns (features). The data is also often not normally distributed.

I have to make a choice between:

  1. Running a small number of trees (i.e. 50 or less) with either complete data or a relatively large and comparative sample
  2. Running several times the number of trees, but with a correspondingly scaled down sample size

There are work-arounds and for any 1 case -- for instance, I can do some ad hoc tests to see which I think will work better, but what I'm wondering is if there is a good theoretical (or robust empirical) reasoning to either guide the choice of approach over the other or to describe the tradeoff being made?

In other words, I'm hoping that someone more comfortable with the math, statistics, and theory underlying this (type of) algorithm can offer some generalizable insight.

$\endgroup$
6
  • $\begingroup$ Mahout has a map reduce implementation. mahout.apache.org/users/classification/…. I don't know what implementation you are using, but in theory each tree can be built independently so the full data set should never need to be in memory. you could also try sub-sampling to remove some of the easily classifiable cases to even out your distribution. $\endgroup$
    – user13684
    Nov 2 '15 at 21:25
  • $\begingroup$ @init-random That is good to know and I will have to check that out! Still, I'm looking for more of a theoretical / statistical answer on this one, so that I'll know what's the best approach in general (or the tradeoffs of either choice) when HPC has been exhausted or when you may not have time to go that route. $\endgroup$
    – Hack-R
    Nov 3 '15 at 1:12
  • $\begingroup$ I can't say for certain, but I would be surprised to find the comparison you are looking for in the literature. You are basically comparing different data sets between (1) and (2) and different models. It would be best keep one stationary and then compare the differences in the moving part. I would think (2) would be your best alternative. See here www-bcf.usc.edu/~gareth/ISL/ISLR%20Fourth%20Printing.pdf p. 320. You could think of (2) as a test/train split for cross validation and choose the model which generalizes best. I would certainly test with OOB error or cross validation. $\endgroup$
    – user13684
    Nov 3 '15 at 2:02
  • $\begingroup$ in the vignette of randomunitforest chapter 5.3 is a piece about incremental learning. It is a useful reading, but only the start. There is little to be found at the moment. $\endgroup$
    – phiver
    Nov 3 '15 at 20:26
  • $\begingroup$ ` Random Forest, Regularized Random Forest, Guided Random Forest` which implementations do you think about ? Python, WEKA or R ? $\endgroup$
    – Qbik
    Apr 27 '16 at 13:57
2
$\begingroup$

I would recommend using a combination of both options #1 and #2.

You could first try tuning your hyper-parameters to find out till what extent could you reduce the number of trees to a point where the random forest model's prediction starts deteriorating on the test set.

This is because changing the value of mtry, the randomly selected number of features for a new tree, is the only meaningful hyper-parameter that should impact accuracy of the model. Since averaging converges as the no. of trees increases, the no. of trees could be reduced to a point where its performance is not impacted as much. Hence, you need to iterate and choose a a limit beyond which very small number of trees may not produce a strong enough ensemble. A random forest needs works best by using more base learners for reducing the variance by averaging each individual tree's output.

It is not clear from your case whether you're using the Random Forest for a a classification or a regression problem. In case this is a classification problem, and if your data-set is imbalanced in terms of ratio of positive vs. negative classes; then you could reduce the size of the training set by under-sampling the majority class to bring it nearer to a 1:1 ratio. Since you have a large number of records, such class based sampling could improve accuracy as well as reduce data size for training.

Additionally, if you've got a fine tuned Random Forest with good performance, then you could also evaluate dropping features that are least important as determined by the algorithm on OOB samples. This would reduce the time taken to train the model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.