I am working on a binary classification project with both continuous and categorical features. I know that the R implementation of RandomForest can handle categorical data passed in as factor type data. Python's scikit-learn implementation requires encoding of categorical data however (e.g. one-hot). I'm curious about the difference in the results I receive using the two implementations, and I'm wondering if anyone knows of a python implementation of RandomForest that can handle categorical data without encoding.
3
-
$\begingroup$ Welcome to Data Science SE. What do you mean "I'm curious about the difference in the results I receive using the two implementations"? $\endgroup$– desertnautOct 22, 2020 at 14:33
-
1$\begingroup$ So its a multilayer question. I'm using RF both as a classifier and also for feature selection with Boruta. My colleague used the R implementation to do feature selection without encoding her categorical data first. She gets somewhat different results than I do with encoded features in python. I figured this has something to do with RFs bias towards categorical features with many options. I'm more comfortable with python, so I was hoping to do some more experimenting with a python implementation, hence the question $\endgroup$– David SteinOct 22, 2020 at 14:40
-
1$\begingroup$ To be more specific, certain high-cardinality categorical features are selected as important in R, but are not selected in python after one-hot encoding $\endgroup$– David SteinOct 22, 2020 at 14:53
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
Because sklearn uses the CART algorithm it cannot accept categorical data as-is (as you have pointed out). There is an existing ticket out to change this.
The issue really should be rephrased as supporting categorical splitting on Decision Trees - not Random Forest, as Random Forrest is simply the ensemble method using these decision trees as fitters.
I don't know of a python package that supports this functionality but I do suspect that it would help increase performance because it would avoid a common pitfall of using random forest on hot-encoded data; see One-Hot Encoding is making your Tree-Based Ensembles worse, here’s why?
In the meantime - I use TargetEncoder when I know I need to use Random Forrest. It allows the categorical data to be encoded in a "smart" fashion without needing to use one-hot encoding.
-
$\begingroup$ This is super helpful, I knew vaguely about issues with RF on one-hot data, but I think I need to read more about it. Definitely going to check out the TargetEncoder $\endgroup$ Oct 22, 2020 at 14:45
$\begingroup$
$\endgroup$
Catboost and LightGBM can handle categorical features. They're based on Decision trees (Random Forest is based on decision trees too), so you can use them (they're usually better than Random Forest), but they use more computational power comparing to Random Forest, yet you still can fine tune them (it's very easy with Catboost, LightGBM needs a little bit of practice), and still benefit from their encoding features, and you'll have less complicated pipeline.
