I do not agree that Bootstrapping is generally superior to using a separate test data set for model assessment.
First of all, it is important here to differentiate between model selection and assessment. In "The Elements of Statistical Learning" (1) the authors put it as following:
Model selection: estimating the performance of different models in
order to choose the best one.
Model assessment: having chosen a final model, estimating its prediction error (generalization error) on new data.
They continue to state:
If we are in a data-rich situation, the best approach for both
problems is to randomly divide the dataset into three parts: a
training set, a validation set, and a test set. The training set is
used to fit the models; the validation set is used to estimate
prediction error for model selection; the test set is used for
assessment of the generalization error of the final chosen model.
Ideally, the test set should be kept in a “vault,” and be brought out
only at the end of the data analysis. Suppose instead that we use the
test-set repeatedly, choosing the model with smallest test-set error.
Then the test set error of the final chosen model will underestimate
the true test error, sometimes substantially.
In line with this the author of "Machine Learning - A Probabilistic Perspective" (2) writes:
Often we use about 80% of the data for the training set, and 20% for the validation set. But if the number of training cases is small, this technique runs into problems, because the mode won’t have enough data to train on, and we won’t have enough data to make a reliable estimate of the future performance. A simple but popular solution to this is to use cross validation (CV)
Since in model selection you are comparing different models (different hyperparameters for a given technique or even different techniques) the absolute level of the prediction error estimate is not that important. As long as the relative error, i.e. how does model A compare to B, is sufficiently well estimated. In contrast, when assessing your final model you would like to obtain an estimate of the absolute prediction error. Also note that in the text quoted by Carlos Mougan the authors write
[...] it can be evaluated on these instances, without the need for a
separate validation set.[...]"
i.e. we are talking validation (model selection) and not testing (models assessment) here. Both, Cross-Validation and Bootstrapping are more common for model selection (i.e. estimate the validation error) but less common for model assessment (i.e. estimate the test error).
To summarize:
- Cross-Validation and Bootstrapping are usually applied when data limited
- Cross-Validation and Bootstrapping are usually applied for model selection/estimation of validation error. They are less common for test error estimates.
Nevertheless Bootstrapping can be used to estimate the validation and even the test error. There are at least 3 ways to apply it (I will now refer to "prediction error" which could either mean "validation error" or "test error" depending on the way you use Bootstrapping):
1. Simple Bootstrap
You produce $B$ Bootstrap samples for your data by randomly selecting $N$ (sample size) data points with repetition. You then train your data on each Bootstrap sample and calculate the prediction error on the whole data set. This is then averaged across all Bootstrap samples $B$. Since the prediction error is based on the whole data set, incl. the examples in the Bootstrap samples, it means that training data is also used to estimate the prediction error. Which can lead to overfitting.
2. Leave-one-out Bootstrap
Here, the approach is similar to Cross-Validation: for a predictor $\hat{f^b}$ trained on Bootstrap sample $b$ the prediction error is estimated based on all data except the examples in the Bootstrap sample $b$. (However, this is not the same as Cross-Validation as Bootstrap samples are produced by drawing with repetition.) This way the problem of overfitting is solved as training and prediction error estimation are based on different sub-data sets for each bootstrap sample.
3. .632 Bootstrap
This approach uses a linear combination of the Leave-one-out Bootstrap prediction error $error_{leave{\text -}one{\text -}out}$ and the simple training error $error_{training}$:
$$error_{.632} = 0.368 \cdot error_{training} + 0.632 \cdot error_{leave{\text -}one{\text -}out} $$
Note that $0.632$ is the probability that when drawing a Bootstrap sample a data point $(x_n,y_n)$ from the overall population is included in that Bootstrap sample. A very casual interpretation of this equation is that you use an estimate that is somewhere between the test error and the leave-one-out Bootstrap error with more weight being put on the latter.
For more details I suggest to read section 7.11 in "The Elements of Statistical Learning - Data Mining, Inference, and Prediction" (1).
(1) "The Elements of Statistical Learning - Data Mining, Inference, and Prediction"; Hastie et al; 2nd Ed
(2) "Machine Learning-A Probabilistic Perspective"; Murphy; 1st Ed