12
$\begingroup$

When saved to disk using cPickle: https://stackoverflow.com/questions/20662023/save-python-random-forest-model-to-file, my random forest is 6.57 GB. 

with open('rforest.cpickle', 'wb') as f:
    cPickle.dump(rforest, f)

I want to use the forest itself to make predictions via a python API hosted on Heroku -- of course, that file size is unacceptable.

Why is the file size so large? There are 500 trees in the ensemble -- all I want to save are the completed trees themselves, since they will be used as prediction. Is it the actual nodes and edges that compose each of the 500 trees that requires nearly 7 GB of space on disk?

I used scikitlearn's randomforestregressor:

def buildForest(self, X_train, y_train):
    rf = RandomForestRegressor(n_estimators=500, verbose=1)
    rf.fit_transform(X_train, y_train)
    return rf

Also, if there is a better way to make my model accessible via API, that would also be good to know.

Update: I reduced it to 100 trees without losing much predictive power, and so now the saved size is 1.3 GB -- much more manageable, but still not great.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Lol and I thought mine was big. My random forest took 330 Mb, I found it weird, thought that was huge size for a file, so I came here and I saw yours takes 6.57 Gb so now I feel better lol thanks. I don't know your number of records but I suppose that will make a difference. I'm also using 500 trees, I tried with 100, 500 and 1000, didn't notice a considerable difference in accuracy so went back to 500. $\endgroup$
    – AndroidMarshmallow
    Jul 7, 2019 at 18:36

4 Answers 4

Reset to default
7
$\begingroup$

The size of each tree depends very much on its depth. Thus, change the maximal depth (max_depth). Try to set it to finite number (as opposed to the default "None") and then try to reduce this number. In addition (or as alternative) try to increase min_samples_split or min_samples_split.

You can also analyze you features and keep only important ones. The simplest way would be to have a look at the clf.feature_importances_ of your forest. (In general, finding important features is an art and science on itself.) Exclude non-relevant features and rebuild the forest.

Improve this answer
$\endgroup$
4
$\begingroup$

Try this：

import pickle
with open('rforest.pickle', 'wb') as f:
    pickle.dump(rforest, f, -1)

Notice: with the parameter "-1" the model file size will largely be reduced.

According to the documentation:

pickle.dump(obj, file[, protocol])

Protocol version 0 is the original ASCII protocol and is backwards compatible with earlier versions of Python. Protocol version 1 is the old binary format which is also compatible with earlier versions of Python. Protocol version 2 was introduced in Python 2.3. It provides much more efficient pickling of new-style classes.

If the protocol parameter is omitted, protocol 0 is used. If protocol is specified as a negative value or HIGHEST_PROTOCOL, the highest protocol version will be used.

Improve this answer
$\endgroup$
1
$\begingroup$

I ran into a similar issue. Even with small tree sizes, I got a file of hundreds of megabytes.

Check if you've set oob_score=True. For large training datasets this can result in a large matrix in oob_decision_function_. I kept the oob_score_, but deleted this matrix. Alternatively, you can set it to False.

Improve this answer
$\endgroup$
1
$\begingroup$

I ran into a similar issue and was surprised to find out that indeed decision trees can easily take a lot of memory (range of MBs) and random forests will easily multiply that in the GB range. Details here: https://stackoverflow.com/a/72633003/4178189

See also some other consideration in this other answer: https://stackoverflow.com/a/72701704/4178189

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.