Multicollinearity in Decision Tree

Question

Can anybody please explain the affect of multicollinearity on Decision Tree algorithms (Classification and regression). I have done some searching but was not able to find the right answer as some say it affects it and others say it doesn't.

Answer 1

Desicion trees make no assumptions on relationships between features. It just constructs splits on single features that improves classification, based on an impurity measure like Gini or entropy. If features A, B are heavily correlated, no /little information can be gained from splitting on B after having split on A. So it would typically get ignored in favor of C.

Of course a single decision tree is very vulnerable to overfitting, so one must either limit depth, prune heavily or preferly average many using an ensemble. Such overfitting problems get worse with many features and possibly also with co-variance but this problem occurs independently from multicolinearity.

While training and predictive power is relatively robust to multicollinearity, feature importance scores will be heavily influenced. So take this into account when trying to interpret the model.

Answer 2

Note that single decision trees are intriniscally greedy algorithms - they will fit on the most effective variable they encounter, leaving other plausible variables out. In the case of multicollinearity, this will likely mean that there is a material probability that the algorithm fits not on the 'right' variable, but on a strong variable correlated with the right variable. If you chose a single decision tree because you wanted something that helped you explain the result, the explanation is thus not likely to be completely coherent or satisfactory.

If you didn't choose your single decision tree in order to get an interpretable result at some level this might not bother you - but if you weren't concerned for interpretation, it seems far more likely that you would have chosen an ensemble tree method or some other black box approach.