1
$\begingroup$

I am experimenting with BagginClassifier, but I fail to get the expected functionality.

Basically, the BagginClassifier should draw (bootstrapping) a new data set with replacement. For example: the following code should generate a new bootstrapped sample of the same size as the original data set:

import sklearn.datasets as ds
import numpy as np


X, y = ds.load_iris().data, ds.load_iris().target

bag = BaggingClassifier(base_estimator=LogisticRegression(), 
                        n_estimators=100,
                        max_samples=1.0,
                        bootstrap=True,
                        n_jobs=1)
bag.fit(X, y)

print X[bag.estimators_samples_[0]].shape
>> 95

(or any other number close to 95).

Naively, I would expect to get the bootstrapped sample of the same size as the original one (150), but with some random repetition of rows. However, I get a smaller sample size with unique rows. That's strange.

What's wrong here?

$\endgroup$
2
$\begingroup$

Found the answer hiding in lines 93-100 in the bagging.py file.
Here is what I understand - the bootstrapping process works in three steps:

  1. Calculate number of samples to train each estimator on (the max_samples variable in the bagging.py code). In your case its $1.0 * x.shape[0] = 150$.

  2. Select with repetition the needed max samples(as calculated in the previous step).
    The selection is done by using randint function, and it generates an array of the x series indices. A given index can appear in this array more than once.

  3. In order to account for indices(samples) that were selected more than one, a weights vector is passed into the base estimator fit function. So, for example, a sample that was selected twice will have $weight * 2$ and it will have the desired impact on the fitting algorithm.

As to my understanding, one can use the estimators_samples_ only to find out which samples were included and not how many times each of them did.

$\endgroup$
  • $\begingroup$ thanks, that totally makes sense to me, however it is a bit frustrating, that the output of sample_ is just a collection of random indices. Why not to create a separate output, smith similar to: .estimators_indices_ to provide us with this functionality and the . .estimators_samples_ will give bootstrapped sample. $\endgroup$ – Arnold Klein Sep 27 '17 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.