Which factors influence the memory consumption?
Is it the number of trees (n_estimators) or rather the number of data records of the training data or something other?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
Both the amount of data and the number of trees in your forest will take up memory storage.
For the amount of data you have, this is obvious why it takes up storage - it’s data, so the more of it you have, the more space it takes up in memory. If your dataset is too large you may not even be able to read it all into memory - it may need to stay on the disk for training (I don’t think scikit supports this).
For the number of trees, each tree that is grown is trained on a subset of random features (so that the forest can avoid variance and avoid all the trees growing the same way). Once the forest is trained, the model object will have information on each individual tree - it’s decision path, split nodes, thresholds etc. This is information, and the more information you have, the more storage it will need. Therefore more trees = more information = more memory usage.
There are other factors that determine how much memory the model object (the trained forest) will take up; for example,
max_depth which sets the maximum number of layers / levels any tree can grow to. If this is large, trees can grow deeper and therefore the model object will be bigger and therefore will need more storage. This shouldn’t be too large though, to avoid overfitting.
The memory usage of the Random Forest depends on the size of a single tree and number of trees. To control the memory size of RF you can:
The default hyper-parameters for Random Forest from scikit-learn are set to build a full tree, which can be very deep in the case of complex datasets.
I was running the experiment where I train RF on Adult Income dataset (~ 3.8MB of data; 32,561 rows and 15 cols). The full tree on this dataset has a depth of about 42 (on average, depends on a bagged sample). The size of the single tree saved to hard drive is about 0.5 MB. And saving the whole Random Forest is about 50 MB.
When I limit the
max_depth of the single tree to
6 then the size of single tree saved to disk was ~
0.01 MB and whole Random Forest saved to disk take ~
0.75 MB, which was
66 times less than RF will full trees!
The interesting part was that I decrease the memory consumption while the performance was up by
The size of single tree in the forest can be also controlled with other parameters:
max_leaf_nodes. To me, '
max_depth is the most intuitive.