I've read other questions regarding if a continuous feature should be converted to categorical or not. But I'm interested in case of tree based classifiers such as Decision Tree, Random Forest, Gradient Boosted etc.

My intuition is that since tree based classifiers try to find the optimal split or the best test at each node, providing a categorical feature would make the splits more accurate than a providing a continuous feature.

My question is, doing the aforementioned pre-processing of data will lead to high accuracy in case of tree based models or the opposite? or it depends on the data?


I actually agreed with the now-deleted answer: you're unlikely to see improvement (at least with any consistency).

As you point out, binning a continuous variable just reduces the allowable split points for a tree model. This reduces the capacity of the overall model. That may be beneficial or detrimental, depending on how it affects the bias-variance tradeoff: I would expect it to help a lot on a fully-grown single decision tree, but probably hurt a little in a well-tuned ensemble of trees.

There are two nice examples in the sklearn docs illustrating their binning preprocessor:

Extending the second one to (1) also include a full decision tree and (2) add binning-then-nonlinear-models produces the following:
Plots showing classifiers on three datasets, with and without binning

At least in these simplistic examples, my suggestions above hold.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.