0
$\begingroup$

Just wanted to check if there are any obvious flaws with a custom encoding idea I have - for categorical features used with RandomForestClassifer or any tree-based classifier.

As all of you would know that sklearn can only handle numerical valued features, categorical features should somehow be encoded to have numerical values. The most recommended encoding techniques on the web are - OneHotEncoding and OrdinalEncoding (and Label Encoding - but a lot of posts say this could make the model flawed, though I wasn't able to find literature that backs this). For the cat feature set in the picture, OneHotEncoding appears to be goto one, as the feature values do not have any inherent order. But since few features have high cardinality and OHE could result in very sparse matrix, we thought of encoding it a bit different as follows:

Cat Features -> LabelEncoder -> Binarize label encoded values

For eg. say, we have a categorical feature called City,

|City|
------
Paris
NY
London
Tokyo

-> After Label Encoding

|City|
------
0
1
2
3

-> After binarization

|City_I|City_II|
----------------
0      | 0
0      | 1
1      | 0
1      | 1

Is there any problem using this custom encoding with tree-based models? Will the performance metrics related to the model help prove that this sort of encoding doesn't tamper with the conceptual soundness of the model?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

There is nothing inherently wrong with any of these encodings. The best way to find out which works best is to use a validation set to verify your encoding methods. Label encoding is usually not preferred for sklearn tree based models because the model treats it as a numerical value and might form a decision tree such as if x>5 go to left tree else go to right tree which does not make any sense. One hot encoding solves this issue but uses alot of memory. One option is to use other models such as xgboost which allow you to pass in integers as categorical variables.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Agreed, but for RandomForest, the split at a node happens based on the max information gain or lowest impurity - so from a list of splits based on a fixed set of features, a specific one is selected. If the label encoded feature is selected as a split condition and the model metrics over train, test, and validation sets are decent - 90% and above, with low variance in each; does that mean that label encoding doesn't tamper with the conceptual soundness of RandomForest?; Apologies if I understood any of these wrong $\endgroup$
    – ranger101
    Feb 18, 2022 at 6:53
0
$\begingroup$

I'm afraid your solution makes no sense. In your example, 'Paris' and 'NY' are equal in City_I, 'NY' and 'Tokyo' in City_II, etc. What would this mean when splitting nodes?

Try instead category_encoders. Also, HistGradientBoostingClassifier has native support for categorical features.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Agreed, one of the most recommended encoding techniques is OneHotEncoding, which creates a sparse matrix, if you consider its output, it'll be roughly like: |City_Ny|City_Paris|City_London|City_Tokyo| 0|1|0|0 1|0|0|0 0|0|1|0 0|0|0|1 So how is this worse that OHE? $\endgroup$
    – ranger101
    Feb 18, 2022 at 9:12
  • $\begingroup$ @ranger101 Every column of one-hot encoded matrix represents a single category. Columns in your matrix cannot be considered individually. Try encoders from category_encoders package. $\endgroup$
    – Sanjar Adilov
    Feb 18, 2022 at 9:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.