2
$\begingroup$

Can anybody please explain the affect of multicollinearity on Decision Tree algorithms (Classification and regression). I have done some searching but was not able to find the right answer as some say it affects it and others say it doesn't.

$\endgroup$
6
$\begingroup$

Desicion trees make no assumptions on relationships between features. It just constructs splits on single features that improves classification, based on an impurity measure like Gini or entropy. If features A, B are heavily correlated, no /little information can be gained from splitting on B after having split on A. So it would typically get ignored in favor of C.

Of course a single decision tree is very vulnerable to overfitting, so one must either limit depth, prune heavily or preferly average many using an ensemble. Such problems get worse with many features and possibly also with co-variance but this problem occurs independently from multicolinearity.

$\endgroup$
3
$\begingroup$

Note that single decision trees are intriniscally greedy algorithms - they will fit on the most effective variable they encounter, leaving other plausible variables out. In the case of multicollinearity, this will likely mean that there is a material probability that the algorithm fits not on the 'right' variable, but on a strong variable correlated with the right variable. If you chose a single decision tree because you wanted something that helped you explain the result, the explanation is thus not likely to be completely coherent or satisfactory.

If you didn't choose your single decision tree in order to get an interpretable result at some level this might not bother you - but if you weren't concerned for interpretation, it seems far more likely that you would have chosen an ensemble tree method or some other black box approach.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.