5
$\begingroup$

Which factors influence the memory consumption?

Is it the number of trees (n_estimators) or rather the number of data records of the training data or something other?

$\endgroup$
  • $\begingroup$ Depends on your dataset a lot, your processor, its cores, ssd/hdd, oob_score set to true or not, n_estimators etc. $\endgroup$ – Aditya Jun 5 '18 at 6:38
  • $\begingroup$ Why processor and hdd/ssd? I mean the RAM-size, sorry I expressed wrong. $\endgroup$ – user43348044 Jun 5 '18 at 6:46
  • $\begingroup$ Computations are done by what(cpu only), you must be kidding me.. hdd/sdd because its faster to access data... $\endgroup$ – Aditya Jun 5 '18 at 6:49
7
$\begingroup$

Both the amount of data and the number of trees in your forest will take up memory storage.

For the amount of data you have, this is obvious why it takes up storage - it’s data, so the more of it you have, the more space it takes up in memory. If your dataset is too large you may not even be able to read it all into memory - it may need to stay on the disk for training (I don’t think scikit supports this).

For the number of trees, each tree that is grown is trained on a subset of random features (so that the forest can avoid variance and avoid all the trees growing the same way). Once the forest is trained, the model object will have information on each individual tree - it’s decision path, split nodes, thresholds etc. This is information, and the more information you have, the more storage it will need. Therefore more trees = more information = more memory usage.

There are other factors that determine how much memory the model object (the trained forest) will take up; for example, max_depth which sets the maximum number of layers / levels any tree can grow to. If this is large, trees can grow deeper and therefore the model object will be bigger and therefore will need more storage. This shouldn’t be too large though, to avoid overfitting.

| improve this answer | |
$\endgroup$
2
$\begingroup$

The main factors are the number of attributes (features) that you have in your dataset and the number of trees in the forest. More attributes means more layers (one per attribute). Do you prune the trees to reduce their depth (and therefore their variance)?

| improve this answer | |
$\endgroup$
  • $\begingroup$ Well not necessarily, since random forest only uses a subset of features, and this is a controllable parameter. $\endgroup$ – Jakub Bartczuk Jun 5 '18 at 9:35
  • 1
    $\begingroup$ Thanks for your helpfull answer! Yes, I prune the tree by setting max_feature, max_depth and min_samples_leaf $\endgroup$ – user43348044 Jun 5 '18 at 19:59
  • $\begingroup$ One question fo Jakun Bartczuk: The Randon Forest only use a subset of featurea, if may_feature is set, right? If not, all trees of the Forest are trained with alle features. $\endgroup$ – user43348044 Jun 5 '18 at 20:01
0
$\begingroup$

The memory usage of the Random Forest depends on the size of a single tree and number of trees. To control the memory size of RF you can:

  • limit the number of trees (the dependency is almost linear),
  • limit the size of a single tree

The default hyper-parameters for Random Forest from scikit-learn are set to build a full tree, which can be very deep in the case of complex datasets.

I was running the experiment where I train RF on Adult Income dataset (~ 3.8MB of data; 32,561 rows and 15 cols). The full tree on this dataset has a depth of about 42 (on average, depends on a bagged sample). The size of the single tree saved to hard drive is about 0.5 MB. And saving the whole Random Forest is about 50 MB.

When I limit the max_depth of the single tree to 6 then the size of single tree saved to disk was ~0.01 MB and whole Random Forest saved to disk take ~ 0.75 MB, which was 66 times less than RF will full trees!

The interesting part was that I decrease the memory consumption while the performance was up by 4% :)

The size of single tree in the forest can be also controlled with other parameters: min_samples_split, min_samples_leaf, min_weight_fraction_leaf, max_features, max_leaf_nodes. To me, 'max_depth is the most intuitive.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.