I am trying to build an ML Classification model on a data set that contains quite a few categorical columns. However, few of them have over 1000 unique values. I am concerned that if I run one-hot encoding or pandas get dummies on them, it will simply result in too many features to work with.

So, I tried to find the top N unique values that account for, say, 90% of the underlying data and group the rest of them under say, 'Other' or 'miscellaneous'. But that's making the 'Other' or 'miscellaneous' value as the most prominent one. I am concerned that this might skew the model/results. Any pointers as to how I should handle such a scenario?

  • $\begingroup$ I'm not sure if this is the right suggestion, but what if you could delete data/rows related to the values with very less frequency? For instance, these values appear only once/twice in your data set? $\endgroup$ Sep 5, 2018 at 6:08
  • $\begingroup$ Yeah, i agree with you. However, if i start doing that with all my categorical columns with too many unique values, im afraid i might be deleting a considerable chunk of the data. $\endgroup$ Sep 5, 2018 at 6:14
  • $\begingroup$ Can you provide a sample ? $\endgroup$
    – n1tk
    Oct 6, 2018 at 23:39

1 Answer 1


If necessary, there are other methods of encoding categorical features:

  • Label encoding (might need some judgement regarding implied ordering)
  • Target encoding
  • Hashing trick

A handy python package is Category Encoders: link

I would suggest first investigating if your model needs categorical variables to be encoded - lightgbm and catboost are examples of ML algorithms which support categorical variables.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.