1
$\begingroup$

I read in comments a recommendation for decision tree´s instead of linear models like neural network, when the dataset has many correlated features. Because to avoid multicollinearity. A similar question is already placed, but not really answered. https://stats.stackexchange.com/questions/137573/do-classification-trees-need-to-consider-the-correlation-between-attributes

or here In supervised learning, why is it bad to have correlated features?

https://www.quora.com/Is-multicollinearity-a-problem-in-decision-trees#:~:text=Decision%20trees%20follow%20the%20non%20parametric%20approach.&text=Though%20single%20tree%20leads%20to,robust%20to%20the%20multi%20collinearity%20.

My problem: I have a dataset of about 30 columns. 10 columns have a high correlation with the target/dependend variable. Data are numerical. I would like to do a prediction (regression model) include all variables if possible?

One big problem is to avoid multicollinearity.

  • Is there a decision tree regression model good when 10 features are high correlated? (if I follow the answers of the links, but there is no really good explanation for that).
  • Is there a scientific or math explanation or recommendation (to use decision tree regression)?
Improve this question
$\endgroup$
11
  • $\begingroup$ problem: I have a dataset of about 30 columns. 10 columns have a high correlation with the target/dependend variable. High correlation has nothing to do with modeling here. $\endgroup$
    – Subhash C. Davar
    Jul 24, 2020 at 23:44
  • $\begingroup$ cor relatedness between probable features is a good basis for classification problem. It is not clear what prompts you to opt for decision tree model. It is based on linear modeling (lm). linear Regression need not be confused with simple linear models that are essentially based on data for correlated features. $\endgroup$
    – Subhash C. Davar
    Jul 25, 2020 at 0:00
  • $\begingroup$ For linear models it´s important to know correlated features. To handle this with VIF. The background of this question was, that I would like to do a prediction of numerical values. However include all variable. Not to kick out any variable with a VIF for linear models (neural net, multipl/regression). At the end I would like to play with all variables (features) to see the behavior, when changing values. Im looking from a engineering perspective. Im not interested just to get a good prediction. Target is a model for "playing" with features to derive information, how component could behav $\endgroup$
    – martin
    Jul 25, 2020 at 14:22
  • $\begingroup$ with that information, im going to change/optimize the structure of a component in engineering software. Data are out of machine tests. I cant see all properties in engineering software. With real world data and software Im optimizing. Hope I could explain it well. That´s the reason, why I want to know: Why can you use decision tree regression without kicking any correlated features like normal regression model or neural net. $\endgroup$
    – martin
    Jul 25, 2020 at 14:31
  • $\begingroup$ In your , step-regression analysis could be useful. Decision tree modeling has a different orientation and a separate purpose. $\endgroup$
    – Subhash C. Davar
    Jul 25, 2020 at 14:33

1 Answer 1

Reset to default
4
$\begingroup$

To answer your questions directly, first:

Is there a decision tree regression model good when 10 features are high correlated?

Yes, definitely. But even better than decision trees, is many decision trees (RandomForest, Gradient Boosting (xGBoost is popular). I think you'd be well served by learning about how decision trees split, and how they naturally deal with collinearity. Maybe try this video Follow the logic until the 2nd tier of splits, and you'll be able to imagine how the correlated variables are suddenly not important because they're relative to the split above them.

Is there a scientific or math explanation or recommendation (to use decision tree regression)?

The math explanation of why collinearity is "bad" for linear models, comes down to the coefficients and how you interpret them. One of the side effects is that they can undermine the statistical significance of a variable, as well as flip their coefficients the wrong direction. It usually doesn't affect the accuracy of the model very much, but most people want linear models so that they can interpret the coefficients (which is totally messed up with collinearity). I suggest reading maybe this article to start.

One of the things that you mentioned, include all variables if possible? is not really something you should be concerned with. The goal of a model is to explain the most, with the least. If you're forcing as many variables as possible into the model, then it's possible that you'll be fooled into thinking a model is good, when in fact it isn't if you were to test it on new data. In fact, sometimes less variables will give you a better model. This is exactly the kind of problem that multicollinearity causes with linear models - that you can't really judge very well what variables are significant or not. Stepwise selection doesn't work very well when there are correlated features.

In general, I think decision trees - especially Random Forests - will be a good start for you. But remember not to force all of the variables into the model just for the sake of it. Experiment with using less variables and manipulating the tree structure such as leaf size and max depth. And as always - test your model on validation data and holdout data so that you don't overfit a model and fool yourself into thinking it's a strong model.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you very much. Funny thing, I already viewed some of your clips in youtube. For this project i don´t reduce variables. (Target is not to reduce variables). It´s an engineering project, at the end I check the behavior of the independent variables to the dependent variable when they got changes. With that information I would like to improve the construction of this automotive component. $\endgroup$
    – martin
    Jul 14, 2020 at 7:23
  • $\begingroup$ So in other words, I could try different decision tree models without reduction of variables. Please correct me if wrong. Thank you. $\endgroup$
    – martin
    Jul 14, 2020 at 7:31
  • $\begingroup$ Yes, a decision tree will naturally "reduce" the variables by simply not using them if they dont provide value by spitting with them. So you can claim you're "using all the variables" but actually some of them are not impacting the model result at all. $\endgroup$
    – Josh
    Jul 14, 2020 at 15:59
  • 2
    $\begingroup$ FYI a good way to measure the behavior of the independent variables (after the model is built) is to use permutation importance. Essentially, it will 1 at a time take a variable and shuffle it, thereby destroying its information. Then the model is scored on holdout and compared to the original model. The relative effect on how bad the model gets when each variable is destroyed will give you a good idea of how important each variable is. This is better than counting on the algorithm itself to provide its importance metric. $\endgroup$
    – Josh
    Jul 14, 2020 at 16:01
  • $\begingroup$ Thank you very much, These steps are very interesting. I didn´t know about permutation importance. $\endgroup$
    – martin
    Jul 14, 2020 at 19:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.