Understanding bootstrapping in bias variance decomposition

Question

I was going through bias and variance tradeoff article and it makes use of bias_variance_decomp function from mlxtend library. This method takes a parameter called num_rounds which is described in API docs as follows:

num_rounds : int (default=200) Number of bootstrap rounds (sampling from the training set) for performing the bias-variance decomposition. Each bootstrap sample has the same size as the original training set.

I was guessing what exactly is it? Given a training set, how many times to run the model on it by sampling the training set? My understanding was, given a training set of say 1 million data points, it will randomly sample "some of" these data points 200 times and train/test model on them. Q1. Am I correct with it? Q2. Also what is the value of "some of" - I mean how it is determined how much data poitns to sample? The doc says "Each bootstrap sample has the same size as the original training set." Does it mean simply select all of 1 million data points? If yes then how is it even sampling (I feel sampling is randomly selecting some data points)? Also this will give same training set each time.

I am sure am missing some basic understanding. What is it?

Answer 1

In bootstrapping, the sampling is done with replacement. So although each sample is the same size as the original training set, it will contain some duplicated instances and omit other instances. This explains how each bootstrapped sample is both the same size as the training set, but they will all be different.

On average, about 63% of instances will be included in each bootstrap sample, and 37% excluded. These excludes instances (the out of bag samples) become the test set for that bootstrapped sample. The overview on this page of the mlxtend documentation tells you a bit more about this.