Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
In scikit-learn documentation and in decision tree learning Wikipedia article there is mention of "There are concepts that are hard to learn because decision trees do not express them easily, such as XOR, parity or multiplexer problems."
I can't recall knowing such problem types.
What are examples (and possibly related datasets) of such problems?
Below is an example of XOR dataset for classification. As you can see, decision trees perform pretty poorly on this dataset. Reason is decision trees splits space into rectangular regions. Therefore they are not pretty good with this kind of distributions.
If you really want to use trees in that sort of situation, it is interesting to use so-called rotation trees. Rotation is about performing PCA (principal components analysis) on input features while learning trees. Using it, decision trees can then build non-rectangular regions.
Also, here is a playground to test gradient boosting algorithms including on XOR dataset. It is really interesting ! You can click on "rotate trees" to activate rotation. Gradient boosting playground
