In random forest algorithm, m (say) number of decision trees are generated using p ( $log_2 n$ +1, where n= number of features) randomly selected features. The label of any sample from test data is chosen by the number of votes by the result of these decision trees. But these decision trees result in different accuracy ( quite obvious). My question is, why should we give same priority to all the decision trees? If a decision tree has higher accuracy in predicting the labels of the test samples, shouldn't we give it more priority than those with lower prediction accuracy ?
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
As you mentioned, each decision tree is trained on p (sometimes sqrt(p) random features). This ensures that each tree is “grown” (trained) differently so that the model 1. Does not overfit the training data (reduces variance) and 2. Generalizes better to new data (reduces bias). Therefore we don’t weigh the trees differently, as this would be similar to having all tress trained on the same features.
You can change the voting threshold however from the standard 50% to anything you want (e.g 40%, 70%, 90%) which will change the precision and recall accuracies of the model.
EDIT: changing the voting threshold means changing the number of trees needed to make a classification. For example, in binary classification most standard random forests require 50% or more of the trees to vote for a class for that class to “win” (be predicted to be that class). But if you change this threshold, to say 70%, then 70% or more of the trees need to vote for the same class for that class to win.
answered May 30, 2018 at 6:07
-
$\begingroup$ Can you please elaborate the voting threshold part a bit more? $\endgroup$– RangerixCommented May 30, 2018 at 6:20
-
$\begingroup$ I saw your edit. Since, features are chosen randomly, it is impossible to say decision trees with higher accuracy will appear more than the decision trees with lower accuracy. If decision trees with lower prediction accuracy appears more, then your idea of voting threshold is not of much use. $\endgroup$– RangerixCommented May 30, 2018 at 8:09
-
$\begingroup$ A random forest is different to that of a single decision tree model; it is an ensemble of decision trees with a few extra architecture features designed to be more robust. For example, the selection of random features for each tree to be trained on. Another is that random forests use bootstrapping, whereby each observation is sampled and replaced, so that on average 1/3 of the total samples are not included in training a tree. As a result, the voting threshold is still a valid parameter to tune. This link may explain more. $\endgroup$ Commented May 30, 2018 at 8:23
$\begingroup$
$\endgroup$
The point is that we can not use test data to choose the best model or give weights to our set of models because test data must be used for the finsl evaluation of our machine learning algorithm. Thus, we have two starategies:
1- Keeping a separate validation data and choose the best model or or give weights to our set of models according to their performance on validation data. This is not the best action because our training data set usually does not include the whole data space, so when we see the new test data which is far from the training set, our model fails. So, our model lacks generalization and is overfitted to validation data and has high variance in terms of expected loss.
2- Training multiple models (trees) and average the results of them which is called bagging. This effectively reduces the variance of the model. In case of random forest which includes random selection of features, the variance is further reduced. This techniques lead to better model generalization (performance on test data).
To cut the long story short, if we choose the best model according to its performance on validation data, we run the risk of over-fitting on validation data. To prevent this problem, we should train multiple models and because we do not know anything about the distribution of test data, we should give equal weigth (priority) to these models while averaging them.
