1
$\begingroup$

Can I do online learning with random forests? I have a few million datapoints and the classifier fails to finish on the cross validation step.

Can i break it up in chunks sequentially?
Current code:

X_train, y_train, X_val, y_val, X_test, y_test = load_dataset()

   print('Planting trees...')
   clf = RandomForestClassifier(
       n_estimators=50,
       max_depth=None,
       min_samples_split=1,
       random_state=0
   )

   print('Growing trees...')
   classifier = clf.fit(X_train, y_train)

   # see how we did
   print('Testing trees...')
   scores = cross_val_score(classifier, X_test, y_test)
   print(scores)
   print('accuracy: %d' % (scores.mean()))

Can I change it to something like:

for chunk in df:
     clf.fit(...)
     cross_validate...
$\endgroup$
2
  • $\begingroup$ Also you have set max depth to None. The trees will be built until node purity is achieved or until all leaves contain less than min_samples_split samples. With a few million data points this can lead to very deep and large trees. $\endgroup$
    – Harpal
    Aug 4, 2016 at 19:50
  • $\begingroup$ I think you have an error in your code.. you first fit the classifier on Xtrain, then you call cross_val_score on Xtest. cross_val_score retrains the classifier on chunks of Xtest! So you're not testing the original classifier trained on Xtrain. To see how it did, just call classifier.predict on Xtest instead of cross_val_score. $\endgroup$
    – stmax
    Aug 5, 2016 at 5:15

1 Answer 1

2
$\begingroup$

There's nothing out of the box that will do true online learning. In order for a scikit-learn algorithm to support online learning it must provide the partial_fit function, which RandomForestClassifier does not. I think the code you have given will just refit the entire forest on the subset of data it is currently looking at.

One alternative you could try is to initialize RandomForestClassifier with warm_start set to True. Then on each subsequent call to fit, the random forest will add a new set of trees to the ensemble. You may have to reduce the number of estimators in the ensemble because the number of estimators you end up with will be the number you initialized with times the number of chunks you loop over.

This won't behave exactly the same way as if you were to train the entire ensemble in a truly online way, but it may be close enough for your purposes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.