how each tree in random forest structured/built?

Question

I'm new to machine learning and I want to use random forest for the problem I have. What I have done so far is I did the 80/20 split of the original data set.

I need to understand what will happen next when building the random forest model. I understand that the next step is taking a random sample (with replacement) from the 80% portion used for training, and use this bootstrapped data set to build a decision tree. Let's assume I have 5 features/columns, and to select a root node, a random subset of the 5 features are selected, and the variable that has the highest Gini index is the root.Let us assume feature 2 is the root.

Next, a random subset of the remaining 4 features are selected to create child nodes for the root node. Let's assume feature 1 and 5 are selected.

My questions are:

1- after obtaining Gini index for feature 1 and 5 and let's assume node 1 has the higher index value, how do I know if node 1 should be the left or right node?

2- does node 5 become the left/right node now? or do we selected a random subset of the remaining 3 features (3, 4, 5), find their Gini index values and the right child becomes the node with the highest Gini index?

Answer 1

Great questions! I'll do my best to answer them:

1. Once you have selected feature 2 as the root and features 1 and 5 as the candidates for the first split, you need to determine whether node 1 should be the left or right node. To make this decision, you will split the data based on the values of feature 2. Any data points with a feature 2 value less than or equal to the threshold value for node 1 will be assigned to the left node, and any data points with feature 2 value greater than the threshold value will be assigned to the right node. Once the data is split, you can calculate the Gini index for each child node using the remaining features.

2. Node 5 is not automatically assigned to the left or right node after node 1 is determined. Instead, you will select a new random subset of the remaining 3 features (3, 4, 5) to create child nodes for node 1. You will repeat the same process as before:

   • Calculate the Gini index for each candidate feature.
   • Select the feature with the highest Gini index as the next node
   • Split the data based on the threshold value of that node.
   • Then, you can calculate the Gini index for each child node using the remaining features.

You will continue this process recursively until you reach a stopping criterion, such as a maximum depth of the tree or a minimum number of samples per leaf node. Each decision tree in the random forest will be trained on a bootstrapped sample of the training data. The final prediction will be the mode (for classification) or mean (for regression) of the predictions from all the trees in the forest.