From The Docs
The RandomForestClassifier is trained using bootstrap aggregation, where each new tree is fit from a bootstrap sample of the training observations z_i = (x_i, y_i)
. The out-of-bag (OOB)
error is the average error for each z_i
calculated using predictions from the trees that do not contain z_i
in their respective bootstrap sample. This allows the RandomForestClassifier to be fit and validated whilst being trained.. So we don't actually need to a Validation dataset or k-fold in this case
Intuitive Understanding
Suppose our training data set is represented by T and suppose data set has M features (or attributes or variables).
T = {(X1,y1), (X2,y2), ... (Xn, yn)} and Xi is input vector {xi1, xi2, ... xiM} and yi is the label (or output or class).
Summary of RF:
Random Forests algorithm is a classifier based on primarily two methods - bagging and random subspace method.
Suppose we decide to have S
number of trees in our forest then we first create S datasets of "same size as original" created from random resampling of data in T
with-replacement (n
times for each dataset). This will result in {T1, T2, ... TS}
datasets. Each of these is called a bootstrap dataset. Due to "with-replacement" every dataset Ti
can have duplicate data records and Ti can be missing several data records from original datasets. This is called Bagging.
Now, RF
creates S
trees and uses m (=sqrt(M) or =floor(lnM+1))
random subfeatures out of M
possible features to create any tree. This is called random subspace method.
So for each Ti
bootstrap dataset you create a tree Ki
. If you want to classify some input data D = {x1, x2, ..., xM}
you let it pass through each tree and produce S
outputs (one for each tree) which can be denoted by Y = {y1, y2, ..., ys}.
Final prediction is a majority vote on this set.
Out-of-bag error:
After creating the classifiers (S trees)
, for each(Xi,yi)
in the original training set i.e. T
, select all Tk
which does not include (Xi,yi)
. This subset, pay attention, is a set of boostrap datasets which does not contain a particular record from the original dataset. This set is called out-of-bag examples. There are n such subsets (one for each data record in original dataset T). OOB classifier is the aggregation of votes ONLY over Tk such that it does not contain (xi,yi).
Out-of-bag estimate for the generalization error is the error rate of the out-of-bag classifier on the training set (compare it with known yi's).
Hope this helps..
Also your n_estimators is Quite High which will lead you to Overffiting