5
$\begingroup$

The algorithm takes quite a long time to train on large data sets with a moderate number of parameters:

https://stats.stackexchange.com/questions/37370/random-forest-computing-time-in-r

https://stackoverflow.com/questions/34997134/random-forest-tuning-tree-depth-and-number-of-trees

https://stackoverflow.com/questions/31278688/how-can-you-reduce-the-default-ntree-500-parameter-passed-to-rf-from-caret

I've been trying to run it on a ~25,000 row data set with 36 predictors and it has been using 6GB of RAM for over 2 hours.

Are there instances where this algorithm is used in production or is being run daily? If so, how does one approach re-training it or optimising it for large data sets?

Improve this question
$\endgroup$
2
  • $\begingroup$ > Online learning. You will train it once, and gradually update in production. Check this cool variant of mondrian forests. $\endgroup$
    – Noah Weber
    Jan 1, 2020 at 11:51
  • $\begingroup$ Try setting n_jobs = -1 if you aren't already. $\endgroup$
    – Brady Gilg
    Jan 2, 2020 at 22:21

2 Answers 2

Reset to default
4
$\begingroup$

For the Random Forests algorithm, the time complexity for building a complete un-pruned tree is $O(m.n\log(n))$, where $n$ is the number of records/instances and $m$ is the number of variables. The algorithm is embarrassingly parallel so in many cases companies with available resources will simply use sufficient compute nodes to enable the model to run in a time that they consider reasonable.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I don't understand the . in your m.n log(n) expression. Is it meant to be multiplication (⋅), or something else? $\endgroup$
    – amalloy
    Jan 2, 2020 at 2:25
  • $\begingroup$ @amalloy yes, it signifies multiplication. $\endgroup$
    – Robert Long
    Jan 2, 2020 at 10:00
0
$\begingroup$

Random Forest is based on bagging and voting.

It means all the individual model has to persist to be used in prediction. Hence increasing the count of estimators will need more RAM.

The good news is that the same property makes it eligible to be trained in parallel if Cores/CPU is available. Please try that.@Brady Gilg has suggested the same.

I have tried ~10 Mn records with estimator count between 15-40 and it needed less than 30 mins. It seems your production system is very light Or something is incorrect with the E2E model.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.