# How do tech-companies employ Random Forest on large data sets?

The algorithm takes quite a long time to train on large data sets with a moderate number of parameters:

https://stats.stackexchange.com/questions/37370/random-forest-computing-time-in-r

https://stackoverflow.com/questions/34997134/random-forest-tuning-tree-depth-and-number-of-trees

https://stackoverflow.com/questions/31278688/how-can-you-reduce-the-default-ntree-500-parameter-passed-to-rf-from-caret

I've been trying to run it on a ~25,000 row data set with 36 predictors and it has been using 6GB of RAM for over 2 hours.

Are there instances where this algorithm is used in production or is being run daily? If so, how does one approach re-training it or optimising it for large data sets?

• > Online learning. You will train it once, and gradually update in production. Check this cool variant of mondrian forests. Jan 1, 2020 at 11:51
• Try setting n_jobs = -1 if you aren't already. Jan 2, 2020 at 22:21

## 2 Answers

For the Random Forests algorithm, the time complexity for building a complete un-pruned tree is $$O(m.n\log(n))$$, where $$n$$ is the number of records/instances and $$m$$ is the number of variables. The algorithm is embarrassingly parallel so in many cases companies with available resources will simply use sufficient compute nodes to enable the model to run in a time that they consider reasonable.

• I don't understand the . in your m.n log(n) expression. Is it meant to be multiplication (⋅), or something else? Jan 2, 2020 at 2:25
• @amalloy yes, it signifies multiplication. Jan 2, 2020 at 10:00

Random Forest is based on bagging and voting.

It means all the individual model has to persist to be used in prediction. Hence increasing the count of estimators will need more RAM.

The good news is that the same property makes it eligible to be trained in parallel if Cores/CPU is available. Please try that.@Brady Gilg has suggested the same.

I have tried ~10 Mn records with estimator count between 15-40 and it needed less than 30 mins. It seems your production system is very light Or something is incorrect with the E2E model.