# ML Models: How to handle categorical feature with over 1000 unique values

I am trying to build an ML Classification model on a data set that contains quite a few categorical columns. However, few of them have over 1000 unique values. I am concerned that if I run one-hot encoding or pandas get dummies on them, it will simply result in too many features to work with.

So, I tried to find the top N unique values that account for, say, 90% of the underlying data and group the rest of them under say, 'Other' or 'miscellaneous'. But that's making the 'Other' or 'miscellaneous' value as the most prominent one. I am concerned that this might skew the model/results. Any pointers as to how I should handle such a scenario?

• I'm not sure if this is the right suggestion, but what if you could delete data/rows related to the values with very less frequency? For instance, these values appear only once/twice in your data set? – Mohit Motwani Sep 5 '18 at 6:08
• Yeah, i agree with you. However, if i start doing that with all my categorical columns with too many unique values, im afraid i might be deleting a considerable chunk of the data. – praveen Sep 5 '18 at 6:14
• Can you provide a sample ? – n1tk Oct 6 '18 at 23:39

A handy python package is Category Encoders: link
I would suggest first investigating if your model needs categorical variables to be encoded - lightgbm and catboost are examples of ML algorithms which support categorical variables.