14
$\begingroup$

Is the random forest implementation in scikit-learn using mean accuracy as its scoring method to estimate generalization error with out-of-bag samples? This is not mentioned in the documentation, but the score() method reports the mean accuracy.

I have a highly unbalanced dataset, and I am using AUC of ROC as my scoring metric in the grid search. Is there a way to tell the classifier to use the same scoring method on the OOB samples as well?

$\endgroup$
  • $\begingroup$ The oob_score parameter of the RandomForestClassifier class doesn't do what you want? $\endgroup$ – Pierre Apr 16 '18 at 16:38
  • $\begingroup$ AFAIK, oob_score report the accuracy. I need to look at the source code again though. $\endgroup$ – darXider Apr 16 '18 at 20:58
  • $\begingroup$ What parameter are you estimating with your grid search? $\endgroup$ – JahKnows Apr 17 '18 at 7:08
  • $\begingroup$ ^ Sorry, but I really do not remember now! I asked this question 1.5 years ago. $\endgroup$ – darXider Apr 17 '18 at 20:30
14
+50
$\begingroup$

In general, the performance of classifiers are compared using accuracy, this is a measure of the number of correctly classified instances divided by the total number of instances. However, from the training data we can get a better approximation of the expected error from our classifier when we are using ensemble learning or bagging techniques.

Out-of-bag error

This metric is the accuracy of examples $x_i$ using all the trees in the random forest ensemble for which it was omitted during training. Thus it kind of acts as a semi-testing instance. You can get a sense of how well your classifier can generalize using this metric.

To implement oob in sklearn you need to specify it when creating your Random Forests object as

from sklearn.ensemble import RandomForestClassifier 
forest = RandomForestClassifier(n_estimators = 100, oob_score = True)

Then we can train the model

forest.fit(X_train, y_train)
print('Score: ', forest.score(X_train, y_train))

Score: 0.979921928817

As expected the accuracy of the model when evaluating the training set is very high. However, this is meaningless because you can very well be overfitting your data and thus your model is rubbish. However, we can use the out-of-bag score as

print(forest.oob_score_)

0.86453272101

This is the accuracy whilst evaluating our instances in the training set using only the trees for which they were omitted. Now let's calculate the score on the testing set as

print('Score: ', forest.score(X_test, y_test))

Score: 0.86517733935

We see that the accuracy measured by oob is very similar to that obtained with the testing set. It thus follows through the theory that the oob accuracy is a better metric by which to evaluate the performance of your model rather than just the score. This is a consequence of bagging models and cannot be done with other types of classifiers.

Calculating oob using different metrics

Yes, you can do this! However, it depends how exactly your code is structured. I am not sure how you can include the oob and AUC all together with the cross_val_score function. However, if you are doing the cross validation folds manually you can do the following, the random forests algorithm in sklearn provides you the decision function of the oob as

print(forest.oob_decision_function_)

The class can then be obtained using

from sklearn import metrics
pred_train = np.argmax(forest.oob_decision_function_,axis=1)

Then we can calculate the AUC by using the following

metrics.roc_auc_score(y_train, pred_train)

0.86217157846471204

$\endgroup$
  • 3
    $\begingroup$ Thanks! I'm aware of how the OOB process works in random forests. I was specifically asking whether RandomForestClassifier can return a OOB score that is NOT accuracy, and the second part of your answer provides a very good hint on how to approach this problem. However, I have to emphasize that one shouldn't use the class labels to calculate the AUC of the ROC curve; rather, the class probabilities should be used directly. So, the correct definition would be pred_train = forest.oob_decision_function_[:, 1]. $\endgroup$ – darXider Apr 17 '18 at 20:24
  • $\begingroup$ @darXider Isn't this shady, since forest.oob_decision_function_.shape[0] == X_train.shape[0], while I'd expect it to be == number of OOB samples, which should be less than the number of samples in X_train? Also, personally I was interested in calculating the logloss, for others that want to do this too, I think pred_train should be = forest.oob_decision_function_ instead. $\endgroup$ – Sander Vanden Hautte Nov 27 '18 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.