3
$\begingroup$

In Scikit-learn's random forest, you can set bootstrap=True and each tree would select a subset of samples to train on. Is there a way to see which samples are used in each tree?

I went through the documentation about the tree estimators and all the attributes of the trees that are made available by Scikit-learn, but none of them seems to provide what I'm looking for.

$\endgroup$
2
  • $\begingroup$ Not possible per my knowledge $\endgroup$ Apr 18, 2021 at 4:20
  • $\begingroup$ The tool is open source so you can modify it on your own computer, in alignment of the license. I have done things similar by changing the code for my own purpose. Check the license. Especially if this is for a business. If this is just for analysis or if there is some production or distribution there may be differences. $\endgroup$
    – Craig
    Apr 18, 2021 at 10:32

2 Answers 2

3
$\begingroup$

I don't think it is possible to get it directly but we may utilize the random seed.

random_stateint, RandomState instance or None, default=None
Controls both the randomness of the bootstrapping of the samples used when building trees (if bootstrap=True)

This is from the RF Github code

def _generate_sample_indices(random_state, n_samples, n_samples_bootstrap):
    """
    Private function used to _parallel_build_trees function."""

    random_instance = check_random_state(random_state)
    sample_indices = random_instance.randint(0, n_samples, n_samples_bootstrap)

    return sample_indices

So, we can get these with a custom code if we fixed the seed above e.g. for 2 Tree,

import pandas as pd, numpy as np
num = 20 # max index
np.random.seed(0) # Fix the seed

sample_1 = np.random.randint(0,num,(1,num))
oob_1 = [elem for elem in np.arange(num) if elem not in sample_1 ]
sample_2 = np.random.randint(0,num,(1,num))
oob_2 = [elem for elem in np.arange(num) if elem not in sample_2 ]

Please verify it with a custom code. I have not verified it.

$\endgroup$
5
  • $\begingroup$ This is what I'm looking for! Thanks! $\endgroup$ Apr 27, 2021 at 6:19
  • $\begingroup$ Please mark the Answer as Accepted. Thanks. $\endgroup$
    – 10xAI
    Apr 27, 2021 at 6:40
  • $\begingroup$ actually I just tried the approach above and it seems like the bootstrapped samples in the random forest are different from the samples generated by randint with the same seed $\endgroup$ Apr 27, 2021 at 8:40
  • $\begingroup$ num = X.shape[0] np.random.seed(27) sample_1 = np.random.randint(0,num,num) sample_2 = np.random.randint(0,num,num) clf1 = DecisionTreeClassifier() clf1.fit(X[sample_1], y[sample_1]) clf2 = DecisionTreeClassifier() clf2.fit(X[sample_2], y[sample_2]) tree1 = clf1.tree_ tree2 = clf2.tree_ print(tree1.value[0]) # list of number of samples from each class at the root node (node 0) np.random.seed(27) rfc = RandomForestClassifier(n_estimators=2, random_state=27) rfc.fit(X,y) rf_tree1 = rfc.estimators_[0].tree_ rf_tree2 = rfc.estimators_[1].tree_ print(rf_tree1.value[0]) $\endgroup$ Apr 27, 2021 at 8:45
  • $\begingroup$ Above is the code that I used to verify the suggested approach. Not sure how to put the code in the comment section properly. $\endgroup$ Apr 27, 2021 at 8:47
1
$\begingroup$

It is possible, actually. The answer is not too different than the one given by @10xAI, but it is not trying to exploit the order of the random seeds implicitly, since it would break for parallel training. So the answer above could maybe only work for trees not trained in parallel. But not sure.

The actual working answer is simple, and it resides in using the random generator stored in each estimator and using it to redo the random sampling.

So, for instance, assume rf is your trained random forest, then it is easy to get both sampled and unsampled indices by importing the appropriate functions and replicating the sampling using the seed in each rf.estimators[0].random_state. For example, to retrieve the lists of sampled and unsampled indices:


import sklearn.ensemble._forest as forest_utils

n_samples = len(Y) # number of training samples

n_samples_bootstrap = forest_utils._get_n_samples_bootstrap(
    n_samples, rf.max_samples
)

unsampled_indices_trees = []
sampled_indices_trees = []

for estimator in rf.estimators_:
    unsampled_indices = forest_utils._generate_unsampled_indices(
        estimator.random_state, n_samples, n_samples_bootstrap)
    unsampled_indices_trees.append(unsampled_indices)

    sampled_indices = forest_utils._generate_sample_indices(
        estimator.random_state, n_samples, n_samples_bootstrap)
    sampled_indices_trees.append(sampled_indices)

estimator is a decision tree in this case, so one can use all the methods to compute custom oob_scores and whatnot.

Hope this helps!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.