# How do tech-companies employ Random Forest on large data sets?

The algorithm takes quite a long time to train on large data sets with a moderate number of parameters:

https://stats.stackexchange.com/questions/37370/random-forest-computing-time-in-r

https://stackoverflow.com/questions/34997134/random-forest-tuning-tree-depth-and-number-of-trees

https://stackoverflow.com/questions/31278688/how-can-you-reduce-the-default-ntree-500-parameter-passed-to-rf-from-caret

I've been trying to run it on a ~25,000 row data set with 36 predictors and it has been using 6GB of RAM for over 2 hours.

Are there instances where this algorithm is used in production or is being run daily? If so, how does one approach re-training it or optimising it for large data sets?

• > Online learning. You will train it once, and gradually update in production. Check this cool variant of mondrian forests. – vienna_kaggling Jan 1 '20 at 11:51
• Try setting n_jobs = -1 if you aren't already. – Brady Gilg Jan 2 '20 at 22:21

For the Random Forests algorithm, the time complexity for building a complete un-pruned tree is $$O(m.n\log(n))$$, where $$n$$ is the number of records/instances and $$m$$ is the number of variables. The algorithm is embarrassingly parallel so in many cases companies with available resources will simply use sufficient compute nodes to enable the model to run in a time that they consider reasonable.
• I don't understand the . in your m.n log(n) expression. Is it meant to be multiplication (⋅), or something else? – amalloy Jan 2 '20 at 2:25