ML Models: How to handle categorical feature with over 1000 unique values

Question

I am trying to build an ML Classification model on a data set that contains quite a few categorical columns. However, few of them have over 1000 unique values. I am concerned that if I run one-hot encoding or pandas get dummies on them, it will simply result in too many features to work with.

So, I tried to find the top N unique values that account for, say, 90% of the underlying data and group the rest of them under say, 'Other' or 'miscellaneous'. But that's making the 'Other' or 'miscellaneous' value as the most prominent one. I am concerned that this might skew the model/results. Any pointers as to how I should handle such a scenario?

Answer 1

If necessary, there are other methods of encoding categorical features:

• Label encoding (might need some judgement regarding implied ordering)
• Target encoding
• Hashing trick

A handy python package is Category Encoders: link

I would suggest first investigating if your model needs categorical variables to be encoded - lightgbm and catboost are examples of ML algorithms which support categorical variables.