OOB decision function doesn't match prediction in scikit-learn RandomForest

Question

I am using a RandomForest for multiclass classification. I would like to use the oob_decision_function to explore precision/recall, but I don't understand the OOB results.

I am using 25,000 trees (n_estimators=25000) and the possible class values are 0,1 or 2.

My results are (truncated to the first 10 rows):

y_train = [1 2 1 1 0 0 2 2 2 2]
y_pred  = [2 2 2 1 0 2 2 2 2 2]

oob_decision_function:
[[ 0.25377529  0.28080796  0.46541674]
 [ 0.32162915  0.3250808   0.35329005]
 [ 0.34463485  0.27709584  0.3782693 ]
 [ 0.31091392  0.2982096   0.39087648]
 [ 0.34932553  0.28632762  0.36434685]
 [ 0.31535905  0.19570567  0.48893528]
 [ 0.25472683  0.35845451  0.38681866]
 [ 0.32521156  0.31721116  0.35757728]
 [ 0.30706625  0.32703203  0.36590172]
 [ 0.29785305  0.22490485  0.4772421 ]]

The dataset is not uniform:

class 0:  32% of samples
class 1:  27% of samples
class 2:  41% of samples

The predictions don't seem to agree with the decision function. See for example the 4th sample: the prediction (y_pred[3]) is class 1, but the OOB values are 0.3109 (class 0), 0.2982 (class 1) and 0.3908 (class 2). Why is the prediction class 1 and not class 2? I thought after 25,000 trees the prediction should closely match the OOB probabilities, or am I not understanding OOB properly?

A few questions:

• how are the OOB values calculated? I thought it was as follows: for each of the 25,000 trees it creates a new bagged training set. Let's say a given sample was not included (ie. out-of-bag) in 6000 of those trees. When the sample is out-of-bag, the current tree is used to predict the class for that sample. So for our given sample, it was estimated 6000 times and let's say the predictions were class 0 (1000 times), class 1 (2000 times) and class 2 (3000 times). Therefore the OOB values would be 1000/6000 (class 0), 2000/6000 (class 1) and 3000/6000 (class 2). Is this correct?

• the OOB values are calculated using the results of the predictions from individual trees (as explained above), but the output predictions (ie. y_pred) are from the final ensemble of trees. Is this correct?

• should the OOB values converge to the cross validation values as the number of trees increases? AFAIK k-fold cross validation splits the data into k subsets and uses the entire forest on each subset.

• is there some sort of weighting being applied (e.g. if my data are not uniformly distributed)?

Answer 1

From your example I understand that $y_{pred}$ are the predictions of the random forest on the training set. If this is the case, $y_{pred}$ would give you training performance whereas the oob values will give you validation performance. Because you are overfitting your training predictions are close to the true values whereas your validation/oob predictions are not.

When bagging, on average each bag contains 2/3 of the samples leaving the remaining 1/3 in the oob. When you use the random forest to predict on the training set, for each sample there will be about 17 000 (~2/3*25000) trees which have seen that datapoint already since it's in their bag and results will not be reliable in terms of generalisation. In other words, when you predict on the training set, 2/3 of the trees will give you training predictions and 1/3 (oob) will give you validation predictions so overall it's like getting training performance.

Answering your last four questions:

• Random forest in sklearn gets the probability of each class for each tree and then averages them across all bags to get the final probabilities. This is usually called 'soft voting', 'hard voting' being the method you describe. See 1.11.2.1. Random Forests

• I haven't seen it stated clearly in the documentation but I would expect that the same method that is used to make general predictions ('soft voting') will be used to get the oob values.

• Yes, cross validation and oob scores should be rather similar since both use data that the classifier hasn't seen yet to make predictions.

• Most sklearn classifiers have a hyperparameter called class_weight which you can use when you have imbalanced data but by default in random forest each sample gets equal weight.

Answer 2

From The Docs

The RandomForestClassifier is trained using bootstrap aggregation, where each new tree is fit from a bootstrap sample of the training observations z_i = (x_i, y_i). The out-of-bag (OOB) error is the average error for each z_i calculated using predictions from the trees that do not contain z_i in their respective bootstrap sample. This allows the RandomForestClassifier to be fit and validated whilst being trained.. So we don't actually need to a Validation dataset or k-fold in this case

(answer isn't strictly acc to your Question ordering)

Intuitive Understanding(The Docs is quite clear if not read below..)

Suppose our training data set is represented by T and suppose data set has M features (or attributes or variables).

T = {(X1,y1), (X2,y2), ... (Xn, yn)} and Xi is input vector {xi1, xi2, ... xiM} and yi is the label (or output or class).

Summary of RF:

Random Forests algorithm is a classifier based on primarily two methods - bagging and random subspace method. 

Suppose we decide to have S number of trees in our forest then we first create S datasets of "same size as original" created from random resampling of data in T with-replacement (n times for each dataset). This will result in {T1, T2, ... TS} datasets. Each of these is called a bootstrap dataset. Due to "with-replacement" every dataset Ti can have duplicate data records and Ti can be missing several data records from original datasets. This is called Bagging. 

Now, RF creates S trees and uses m (=sqrt(M) or =floor(lnM+1)) random subfeatures out of M possible features to create any tree. This is called random subspace method. 

So for each Ti bootstrap dataset you create a tree Ki. If you want to classify some input data D = {x1, x2, ..., xM} you let it pass through each tree and produce S outputs (one for each tree) which can be denoted by Y = {y1, y2, ..., ys}. Final prediction is a majority vote on this set. 

Out-of-bag error:

After creating the classifiers(S trees), for each(Xi,yi)in the original training set i.e. T, select all Tkwhich does not include (Xi,yi). This subset, pay attention, is a set of boostrap datasets which does not contain a particular record from the original dataset. This set is called out-of-bag examples. There are n such subsets (one for each data record in original dataset T). OOB classifier is the aggregation of votes ONLY over Tk such that it does not contain (xi,yi). 

Out-of-bag estimate for the generalization error is the error rate of the out-of-bag classifier on the training set (compare it with known yi's).

Also your n_estimators is Quite High which will lead you to Overffiting

Also RF isn't a good option when we have severe imbalanceness As Random forests are built on decision trees, and decision trees are sensitive to class imbalance. Therefore each tree will be biased in the same direction and magnitude (on average) by class imbalance.

Just a side note,

Bagging and Boosting might sound similar but aren't similar.. Xgboost can take care of Imbalanceness whereas RF's don't..

Hope it helps..