1
$\begingroup$

First, I am confused whether at each node in all the trees, do we randomly pick features from the lot to be pitted for best split or does each tree get a random subset of feature and then all the nodes in that tree have to work with those features?

Now, regardless of the case whether we pick random features at tree or node level, why do choose random subset of data for each tree? Because if the features are different at tree/node level, the trees will be different. Why do we need to add another level and make the distribution of data also random to the trees and not just feed the complete original dataset to the trees.

$\endgroup$
4
  • $\begingroup$ Have you read the original paper where random forest is introduced? $\endgroup$ – WoofDoggy Apr 25 '19 at 15:01
  • $\begingroup$ @WoofDoggy No, what do you mean? So, amateurs cannot venture into this field if they don't go through original published papers. $\endgroup$ – pragun Apr 26 '19 at 14:04
  • $\begingroup$ Scientific papers are not that hard to read. The advantage is that you are presented with the initial idea which is not consumed or reinterpreted by others. The knowledge you get from books may sometimes be interpreted in the wrong way $\endgroup$ – WoofDoggy Apr 26 '19 at 14:30
  • $\begingroup$ I understand that but a lot of papers on statistics field involve cryptic mathematical notations and equations which deters people like me away from data science. I am sure you would have come across such. I am always on the lookout for something which can be explained in layman terms with blending mathematics. $\endgroup$ – pragun Apr 26 '19 at 14:56
1
$\begingroup$

First, I am confused whether at each node in all the trees, do we randomly pick features from the lot to be pitted for best split or does each tree get a random subset of feature and then all the nodes in that tree have to work with those features?

m-features are chosen randomly at each split of the tree, not at the creation of the tree itself.

Now, regardless of the case whether we pick random features at tree or node level, why do choose random subset of data for each tree? Because if the features are different at tree/node level, the trees will be different. Why do we need to add another level and make the distribution of data also random to the trees and not just feed the complete original dataset to the trees?

This is exactly what makes random forests unique compared to other algorithms. Indeed, we sample with replacement (bootstrap) from the training dataset and each bootstrap sample is unique to each tree (which you appear to understand).

However, we need to consider how the final predictions come out of a random forest in the first place. Recall that the final prediction for an observation $y_i$ will be the average predicted probability among all of the trees in the forest. This is supposed to work because, under the (crucial) assumption of zero/negative correlation between the trees, we know that the sample mean has variance that goes to zero as n goes to infinity. This is one key element of random forests; if I have a bunch of overfit trees (low bias, high variance) I can simply get rid of this high variance for each tree by combining the predictions of each individual tree and averaging. The result is hopefully low bias, and low enough variance for my final, combined model.

However, if there is positive correlation among the trees, the variance of the sample mean actually increases (as $ n \rightarrow \infty$) and if n is large enough, the covariance between the trees actually starts to dominate the variance of each individual tree itself. What this results in is a decrease in predictive performance compared to just using a single (or relatively few) tree(s). After all, the whole point is to fit low bias, high variance base learners and to then reduce the variance by averaging. If we now cannot reduce the variance due to correlation between base learners (and start increasing it instead!), our model is likely to be severely overfit, even moreso than using a comparatively less overfit base learner.

So the question becomes...how do we reduce the correlation between the trees? One solution is to introduce randomness not only through the data sample itself (i.e. bootstrap/sample with replacement) but also through randomly selecting predictors at each split of a node. By doing this, we hopefully force each tree to learn something different about the data. Indeed, if only a few predictors are useful than chances are, all of the trees will pick up on this and stick to using these few predictors over and over again. The result is that all of the trees have learned exactly the same thing, substantial correlation among the trees, and not much purpose to using the trees together.

Bagging (which is what you refer to as the first level of randomness) and other ensembling based methods work best when we have what people call "diverse base learners". This means that in general, if we can find many models that are similar in performance but uncorrelated (that is, they have learned different, but equally important parts of the data) then combining the models in some fashion will almost certainly be better than using either one alone. Creating model diversity is difficult but some common ways are;

1) Do something similar to random forests; give each base learner a different set of features to use.
2) Use different algorithms that hopefully learn different parts of the data due to the differences in how they are fit; example: random forest + neural network + gradient boosting, etc.

$\endgroup$
0
$\begingroup$

The complete tree gets a random subset of features, then works with it at a tree level.

Correction:

Correct answer

As correctly stated in this answer, every node gets its own random selection of features.

The explanation to what you ask in the second paragraph is that the trees are not globally optimal but locally optimal.

The random subset is selected because is more likely to find a global optimum if a random selection is done.

As an example:

  • Suppose we have a dataset with Age, Gender, Salary, Marital Status and Nationality.
  • The optimal tree is always unknown, but let's suppose that we know it and is made from the path Age-Gender-Salary.
  • What Random Forest does is random select between Age, Gender, Salary, Marital Status and Nationality. What a normal decision tree does is to select all the features to make the tree.
  • The decision tree selects this tree: Salary-Marital Status-Age ALWAYS. Why is it possible to have this decision tree? Because salary is the local optimum (the best variable at the first step), but Salary-first-then-others could not be the best combination.
  • The Random Forest selects many possible combinations of the variables, in which we could find Age-Gender-Salary which is the optimal.

The way Random Forest works is to try many combinations of variables, one of them could be the optimal.

$\endgroup$
6
  • 2
    $\begingroup$ I found a similar question here datascience.stackexchange.com/questions/18199/… and the answer says at each split in all the trees, variables are selected randomly from the whole feature set. $\endgroup$ – pragun Apr 26 '19 at 14:07
  • $\begingroup$ You are right! I will clarify in my answer. $\endgroup$ – Juan Esteban de la Calle Apr 26 '19 at 14:16
  • $\begingroup$ I am sorry, i wasn't able to get along with your explanation of points break down.Can you please go through your answer again and check if it might need some clarity. $\endgroup$ – pragun Apr 26 '19 at 14:18
  • $\begingroup$ Yes, what I tried to say is that the randomness in the selection of features allows the algorithm to reach combinations of variables which would unaccesible if it used the same variables always. $\endgroup$ – Juan Esteban de la Calle Apr 26 '19 at 14:22
  • $\begingroup$ Agreed! But as i said in my question, in random forest, we can make different trees with same original dataset because we are picking features randomly at every split. So, why do we need different randomized data for each tree. I believe it makes randomness at level 2( one with features and 2nd with dataset) $\endgroup$ – pragun Apr 26 '19 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.