RF and DT overfitting

Question

I am new with Machine Learning and I started with some lessons in Kaggle. There, I learnt how to use DecisionTreeRegressor() and RandomForestRegressor() from sklearn.

---

However, I cannot really understand how I can verify that my explanatory variables do not overfit the model. For example, the lessons included evaluation with the use of Mean Absolute Error. MAE and MRSE can evaluate whether my Decision Tree depth is optimal or not, but not if my explanatory data are even relevant.

---

I come from Economics, so I am used to deal with such problems using diagnostics or $R^2$. Is there any equivalent to $R^2$ benchmark to determine whether my explanatory variables are overfitting my model or not?

Answer 1

I think you can perform Predictor Importance test and see which are the variable explaining the most.

There is this package named Boruta, you can go through the link for implementation in python.

You can eliminate the variables which are highly correlated. For example if you have age as the target variable and you have DOB as a feature then it makes no sense to build a model. So, you need to make sure to eliminate the variable which are highly correlated to the target variable.

In my Scenario I had this following visualization

As you can see the 2 variables(underlined with red dash) are highly correlated with the target variable, before removing these variables the MAE was 0.9(approx) after removing those features(Backward Stepwise Elimination) and the MAE was 3.5(approx) but that is the actual error. Currently working on getting some external features to explain the data and to improve the accuracy. Every time it is not about accuracy/error rate of the model, it is also about how good our model could be generalized and robust our model should be.

To check if the data is overfitting, then I tried testing it by taking those 2 variables and try modelling and the MAE was 1.6(approx) from this we can understand that these 2 variables explain the most.

So, try applying and see how the features are correlated with the target variable.

One of the methods used to address over-fitting in decision tree is called pruning which is done after the initial training is complete. In pruning, you trim off the branches of the tree, i.e., remove the decision nodes starting from the leaf node such that the overall accuracy is not disturbed. This is done by segregating the actual training set into two sets: training data set, D and validation data set, V. Prepare the decision tree using the segregated training data set, D. Then continue trimming the tree accordingly to optimize the accuracy of the validation data set, V.

You can go thorough this link, about how we can avoid over fitting by tuning the parameters.

Answer 2

Yes, and that benchmark is called validation data. The idea is to split your data in some training data, which is what you are going to fit to your models in order to estimate the parameters in them (and make them learn), and some validation data, which is the data you are going to use in order to evaluate your model.

What you can do is compute the error on your validation data (which the models haven't been trained on) for the different models you want to use, and you would like to keep the model with the lowest validation error.

Answer 3

I think what you are trying to ask is whether your variables are important or not. There is no thing as the variables are overfitting to the model, however you can overfit to the model by tuning and minimizing bias to the very end.

In models such as Decision Trees and Multiple Decision Trees Ensembles (Random Forest), you can calculate variable importance by using entropy measure of the variables. This will lead you to understand which variables are actually important.

Lastly, in models such as random forest the intuition is to have multiple decision trees where at each splitting node having different variables across many trees. Most of the time you would not even use all the variables you have in your data set in a single tree. This enable to decrease your model's variance while not affecting your bias.