4
$\begingroup$

The algorithm takes quite a long time to train on large data sets with a moderate number of parameters:

https://stats.stackexchange.com/questions/37370/random-forest-computing-time-in-r

https://stackoverflow.com/questions/34997134/random-forest-tuning-tree-depth-and-number-of-trees

https://stackoverflow.com/questions/31278688/how-can-you-reduce-the-default-ntree-500-parameter-passed-to-rf-from-caret

I've been trying to run it on a ~25,000 row data set with 36 predictors and it has been using 6GB of RAM for over 2 hours.

Are there instances where this algorithm is used in production or is being run daily? If so, how does one approach re-training it or optimising it for large data sets?

$\endgroup$
2
  • $\begingroup$ > Online learning. You will train it once, and gradually update in production. Check this cool variant of mondrian forests. $\endgroup$ – vienna_kaggling Jan 1 '20 at 11:51
  • $\begingroup$ Try setting n_jobs = -1 if you aren't already. $\endgroup$ – Brady Gilg Jan 2 '20 at 22:21
3
$\begingroup$

For the Random Forests algorithm, the time complexity for building a complete un-pruned tree is $O(m.n\log(n))$, where $n$ is the number of records/instances and $m$ is the number of variables. The algorithm is embarrassingly parallel so in many cases companies with available resources will simply use sufficient compute nodes to enable the model to run in a time that they consider reasonable.

$\endgroup$
2
  • $\begingroup$ I don't understand the . in your m.n log(n) expression. Is it meant to be multiplication (), or something else? $\endgroup$ – amalloy Jan 2 '20 at 2:25
  • $\begingroup$ @amalloy yes, it signifies multiplication. $\endgroup$ – Robert Long Jan 2 '20 at 10:00
0
$\begingroup$

Random Forest is based on bagging and voting.

It means all the individual model has to persist to be used in prediction. Hence increasing the count of estimators will need more RAM.

The good news is that the same property makes it eligible to be trained in parallel if Cores/CPU is available. Please try that.@Brady Gilg has suggested the same.

I have tried ~10 Mn records with estimator count between 15-40 and it needed less than 30 mins. It seems your production system is very light Or something is incorrect with the E2E model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.