How are Decision Trees averaged in Random Forest?

Question

We all know that a Random Forest is an ensemble of Decision Trees, whose results are averaged.

Every source I find simply talks about "averaging trees", but how does this "averaging of trees" happens? Can anyone exaplain it in detail?

Answer 1

It depends on the type of variable that the Random Forest is predicting, and perhaps on the specific implementation of Random Forest. Following is an overview of the simplest techniques.

Continuous Target

In the case of a continuous target variable $y$, each of the trees in an ensemble will generate a prediction $\hat{y}_i$. The most naive way to combine the results from trees in an ensemble is to take the mean of all predictions.

Suppose you have an ensemble of 10 trees. Then the combined prediction $\hat{y}$ would be computed as $$\hat{y} = \frac{1}{10} \sum^{10}_{i=1} \hat{y}_i$$

This is the approach taken by forest regressors in scikit-learn. Notice the lines

    # Parallel loop
    lock = threading.Lock()
    Parallel(n_jobs=n_jobs, verbose=self.verbose,
             **_joblib_parallel_args(require="sharedmem"))(
        delayed(_accumulate_prediction)(e.predict, X, [y_hat], lock)
        for e in self.estimators_)

    y_hat /= len(self.estimators_)

The "Parallel Loop" part sums up the predictions made by each tree in the forest, and the last line divides by the total number of trees so that y_hat is the mean of the predictions from each tree.

An alternative approach is to take a weighted average of the predictions $$\hat{y} = \sum^{10}_{i=1} \omega_i \hat{y}_i$$

Where each tree has a weight $\omega_i$. The weight could be determined, for example, by the tree's performance. This approach might slightly improve accuracy in some cases, but it's easier to overfit.

Categorical Target

In the case where your random forest is predicting the value of a categorical variable, you can allow each tree to act like a member of a "committee" and cast a vote.

Suppose a categorical variable has three possible values, $A$, $B$, or $C$, and your random forest has 10 trees. You generate a prediction with each tree, and that prediction counts as one vote.

For example, suppose 5 of your trees predict $A$, 3 of them predict $B$, and 2 of them predict $C$. The combined prediction will be $A$.

Notice that you may run into cases where there is a tie. Suppose 5 of your trees predict $A$, and the other five predict $B$. Which prediction do you go with?

One option is to predict the majority class in the case of a tie. In the example above, if the training data contains more $B$ examples than $A$ examples, then we would predict $B$.

Another option for resolving ties is to weight votes by the accuracy of the tree. Suppose again that you have five trees that vote for $A$ and five trees that vote for $B$. If you observe that, on average, the trees voting for $B$ are more reliable, you may give their votes extra weight and predict $B$.

Hopefully that helps!