3
$\begingroup$

I am trying to build an ML Classification model on a data set that contains quite a few categorical columns. However, few of them have over 1000 unique values. I am concerned that if I run one-hot encoding or pandas get dummies on them, it will simply result in too many features to work with.

So, I tried to find the top N unique values that account for, say, 90% of the underlying data and group the rest of them under say, 'Other' or 'miscellaneous'. But that's making the 'Other' or 'miscellaneous' value as the most prominent one. I am concerned that this might skew the model/results. Any pointers as to how I should handle such a scenario?

$\endgroup$
  • $\begingroup$ I'm not sure if this is the right suggestion, but what if you could delete data/rows related to the values with very less frequency? For instance, these values appear only once/twice in your data set? $\endgroup$ – Mohit Motwani Sep 5 '18 at 6:08
  • $\begingroup$ Yeah, i agree with you. However, if i start doing that with all my categorical columns with too many unique values, im afraid i might be deleting a considerable chunk of the data. $\endgroup$ – praveen Sep 5 '18 at 6:14
  • $\begingroup$ Can you provide a sample ? $\endgroup$ – n1tk Oct 6 '18 at 23:39
2
$\begingroup$

If necessary, there are other methods of encoding categorical features:

  • Label encoding (might need some judgement regarding implied ordering)
  • Target encoding
  • Hashing trick

A handy python package is Category Encoders: link

I would suggest first investigating if your model needs categorical variables to be encoded - lightgbm and catboost are examples of ML algorithms which support categorical variables.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.