I am new with Machine Learning and I started with some lessons in Kaggle. There, I learnt how to use DecisionTreeRegressor() and RandomForestRegressor() from sklearn.

However, I cannot really understand how I can verify that my explanatory variables do not overfit the model. For example, the lessons included evaluation with the use of Mean Absolute Error. MAE and MRSE can evaluate whether my Decision Tree depth is optimal or not, but not if my explanatory data are even relevant.

I come from Economics, so I am used to deal with such problems using diagnostics or $R^2$. Is there any equivalent to $R^2$ benchmark to determine whether my explanatory variables are overfitting my model or not?

  • $\begingroup$ How did you use diagnostics and R-squared to check overfitting in Economics? $\endgroup$ – Michael M Apr 20 '18 at 9:43

I think you can perform Predictor Importance test and see which are the variable explaining the most.

There is this package named Boruta, you can go through the link for implementation in python.

You can eliminate the variables which are highly correlated. For example if you have age as the target variable and you have DOB as a feature then it makes no sense to build a model. So, you need to make sure to eliminate the variable which are highly correlated to the target variable.

In my Scenario I had this following visualization

Predictor Importance

As you can see the 2 variables(underlined with red dash) are highly correlated with the target variable, before removing these variables the MAE was 0.9(approx) after removing those features(Backward Stepwise Elimination) and the MAE was 3.5(approx) but that is the actual error. Currently working on getting some external features to explain the data and to improve the accuracy. Every time it is not about accuracy/error rate of the model, it is also about how good our model could be generalized and robust our model should be.

To check if the data is overfitting, then I tried testing it by taking those 2 variables and try modelling and the MAE was 1.6(approx) from this we can understand that these 2 variables explain the most.

So, try applying and see how the features are correlated with the target variable.

One of the methods used to address over-fitting in decision tree is called pruning which is done after the initial training is complete. In pruning, you trim off the branches of the tree, i.e., remove the decision nodes starting from the leaf node such that the overall accuracy is not disturbed. This is done by segregating the actual training set into two sets: training data set, D and validation data set, V. Prepare the decision tree using the segregated training data set, D. Then continue trimming the tree accordingly to optimize the accuracy of the validation data set, V.

You can go thorough this link, about how we can avoid over fitting by tuning the parameters.

  • 1
    $\begingroup$ Would you please explain how you think identifying and removing the correlated features would help here? As far as I know Tree-based methods are robust to feature correlations, e.g. look here: medium.com/data-design/…. In case the model overfits, there are other recommend ways to reduce it. $\endgroup$ – TwinPenguins Apr 20 '18 at 11:47
  • $\begingroup$ Hey man, so what it means is if there is any correlation with-in the features then it is handled well(model is robust) but when it comes to the correlation with respect to target variable is very important. The reason why it is important in this scenario is, if you have a highly correlated then the model which we built is not robust and it is highly dependent on these 2 features. There are many methods but here in this scenario he was asking about correlation, so answered it in those lines. The other answer suggests about validation set there are many other ways to find it. $\endgroup$ – Toros91 Apr 20 '18 at 12:22
  • 1
    $\begingroup$ Hey. Thanks for your the explanation. I learned about the Boruta btw. The reason I commented was that I still do not think your answer is relevant to what the user asked although he/she may not even realizing it. The question is about overfitting, and NOT correlation. Yes, correlation can lead to overfitting, I am with you and kind of relevant you brought it up. That would be better and more important if you could talk about more relevant methods to avoid overfitting esp. when the models of interest here are Tree-based, which are quite robust to correlations. $\endgroup$ – TwinPenguins Apr 20 '18 at 12:52
  • $\begingroup$ @MajidMortazavi: sure, would update my answer with relevant methods for avoid over fitting the tree based models. Do you really think that my answer is completely irrelevant? $\endgroup$ – Toros91 May 4 '18 at 3:49

Yes, and that benchmark is called validation data. The idea is to split your data in some training data, which is what you are going to fit to your models in order to estimate the parameters in them (and make them learn), and some validation data, which is the data you are going to use in order to evaluate your model.

What you can do is compute the error on your validation data (which the models haven't been trained on) for the different models you want to use, and you would like to keep the model with the lowest validation error.

  • $\begingroup$ Ok, but this does not seem to solve anything if my explanatory variables are overfitting the dependent (i.e. it does not enable any research to achieve a parsimonious model). Do I miss something? :/ $\endgroup$ – Commissar Vasili Karlovic Apr 20 '18 at 8:32
  • 2
    $\begingroup$ Your explanatory variables do not overfit the dependent. Models overfit, variables don't. What do you mean by a parsimonious model? $\endgroup$ – David Masip Apr 20 '18 at 8:42
  • $\begingroup$ The complexity of decision trees and random forests are controlled via their parameter specifications - e.g. max # trees, max tree depth, etc. So you can achieve a "parsimonious model" via an appropriate parameter specification. To understand the "relevance" of each explanatory variable in predicting the response, you can extract variable importance measures from the models. $\endgroup$ – bradS May 4 '18 at 7:26

I think what you are trying to ask is whether your variables are important or not. There is no thing as the variables are overfitting to the model, however you can overfit to the model by tuning and minimizing bias to the very end.

In models such as Decision Trees and Multiple Decision Trees Ensembles (Random Forest), you can calculate variable importance by using entropy measure of the variables. This will lead you to understand which variables are actually important.

Lastly, in models such as random forest the intuition is to have multiple decision trees where at each splitting node having different variables across many trees. Most of the time you would not even use all the variables you have in your data set in a single tree. This enable to decrease your model's variance while not affecting your bias.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.