How to perform bootstrap validation?

Question

I am working on a binary classification problem.

I ran cross-validation and grid-search on train data. Later I validated the model on my test data as shown below

logreg=LogisticRegression(random_state=41)
logreg_cv=GridSearchCV(logreg,op_param_grid,cv=10,scoring='f1')
logreg_cv.fit(X_train_std,y_train)
y_pred = logreg_cv.predict(X_test_std)

But I came across the term bootstrap validation on how it can be used to avoid test data poor performance due to split imbalances (in test data).

Can someone guide me on what is bootstrap validation and how to implement this?

Answer 1

I do not agree that Bootstrapping is generally superior to using a separate test data set for model assessment.

First of all, it is important here to differentiate between model selection and assessment. In "The Elements of Statistical Learning" (1) the authors put it as following:

Model selection: estimating the performance of different models in order to choose the best one.

Model assessment: having chosen a final model, estimating its prediction error (generalization error) on new data.

They continue to state:

If we are in a data-rich situation, the best approach for both problems is to randomly divide the dataset into three parts: a training set, a validation set, and a test set. The training set is used to fit the models; the validation set is used to estimate prediction error for model selection; the test set is used for assessment of the generalization error of the final chosen model. Ideally, the test set should be kept in a “vault,” and be brought out only at the end of the data analysis. Suppose instead that we use the test-set repeatedly, choosing the model with smallest test-set error. Then the test set error of the final chosen model will underestimate the true test error, sometimes substantially.

In line with this the author of "Machine Learning - A Probabilistic Perspective" (2) writes:

Often we use about 80% of the data for the training set, and 20% for the validation set. But if the number of training cases is small, this technique runs into problems, because the mode won’t have enough data to train on, and we won’t have enough data to make a reliable estimate of the future performance. A simple but popular solution to this is to use cross validation (CV)

Since in model selection you are comparing different models (different hyperparameters for a given technique or even different techniques) the absolute level of the prediction error estimate is not that important. As long as the relative error, i.e. how does model A compare to B, is sufficiently well estimated. In contrast, when assessing your final model you would like to obtain an estimate of the absolute prediction error. Also note that in the text quoted by Carlos Mougan the authors write

[...] it can be evaluated on these instances, without the need for a separate validation set.[...]"

i.e. we are talking validation (model selection) and not testing (models assessment) here. Both, Cross-Validation and Bootstrapping are more common for model selection (i.e. estimate the validation error) but less common for model assessment (i.e. estimate the test error).

To summarize:

1. Cross-Validation and Bootstrapping are usually applied when data limited
2. Cross-Validation and Bootstrapping are usually applied for model selection/estimation of validation error. They are less common for test error estimates.

Nevertheless Bootstrapping can be used to estimate the validation and even the test error. There are at least 3 ways to apply it (I will now refer to "prediction error" which could either mean "validation error" or "test error" depending on the way you use Bootstrapping):

1. Simple Bootstrap

You produce $B$ Bootstrap samples for your data by randomly selecting $N$ (sample size) data points with repetition. You then train your data on each Bootstrap sample and calculate the prediction error on the whole data set. This is then averaged across all Bootstrap samples $B$. Since the prediction error is based on the whole data set, incl. the examples in the Bootstrap samples, it means that training data is also used to estimate the prediction error. Which can lead to overfitting.

2. Leave-one-out Bootstrap

Here, the approach is similar to Cross-Validation: for a predictor $\hat{f^b}$ trained on Bootstrap sample $b$ the prediction error is estimated based on all data except the examples in the Bootstrap sample $b$. (However, this is not the same as Cross-Validation as Bootstrap samples are produced by drawing with repetition.) This way the problem of overfitting is solved as training and prediction error estimation are based on different sub-data sets for each bootstrap sample.

3. .632 Bootstrap

This approach uses a linear combination of the Leave-one-out Bootstrap prediction error $error_{leave{\text -}one{\text -}out}$ and the simple training error $error_{training}$: $$error_{.632} = 0.368 \cdot error_{training} + 0.632 \cdot error_{leave{\text -}one{\text -}out} $$

Note that $0.632$ is the probability that when drawing a Bootstrap sample a data point $(x_n,y_n)$ from the overall population is included in that Bootstrap sample. A very casual interpretation of this equation is that you use an estimate that is somewhere between the test error and the leave-one-out Bootstrap error with more weight being put on the latter.

For more details I suggest to read section 7.11 in "The Elements of Statistical Learning - Data Mining, Inference, and Prediction" (1).

---

(1) "The Elements of Statistical Learning - Data Mining, Inference, and Prediction"; Hastie et al; 2nd Ed

(2) "Machine Learning-A Probabilistic Perspective"; Murphy; 1st Ed

Answer 2

I am just gonna copy from Oreilly's book Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition:

With bagging, some instances may be sampled several times for any given predictor, while others may not be sampled at all. By default a BaggingClassifier samples m training instances with replacement (bootstrap=True), where m is the size of the training set. This means that only about 63% of the training instances are sampled on average for each predictor.6 The remaining 37% of the training instances that are not sampled are called out-of-bag (oob) instances. Note that they are not the same 37% for all predictors.

Since a predictor never sees the oob instances during training, it can be evaluated on these instances, without the need for a separate validation set. You can evaluate the ensemble itself by averaging out the oob evaluations of each predictor.

In Scikit-Learn, you can set oob_score=True when creating a BaggingClassifier to request an automatic oob evaluation after training. The following code demonstrates this. The resulting evaluation score is available through the oob_score_ variable:

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier


>>> bag_clf = BaggingClassifier(
...     DecisionTreeClassifier(), n_estimators=500,
...     bootstrap=True, n_jobs=-1, oob_score=True)
...
>>> bag_clf.fit(X_train, y_train)
>>> bag_clf.oob_score_
0.90133333333333332

According to this oob evaluation, this BaggingClassifier is likely to achieve about 90.1% accuracy on the test set. Let’s verify this:

>>> from sklearn.metrics import accuracy_score
>>> y_pred = bag_clf.predict(X_test)
>>> accuracy_score(y_test, y_pred)
0.91200000000000003