I'm doing some data processing and wondering what is the best practice for dealing with categorical columns that has a value counts plot looking like the below (these are one-hot-encoded at a later stage). Should the rarely represented variables be dropped? What is a typical limit in terms of value count % that decides if they should be dropped? Should I group the categories between for example "Tug" to "Naval" as "Other"?

I realize that how the data is processed is dependent on the underlying "meaning" of the data. Nevertheless, would love some general comments on best practices for situations like this.

enter image description here

  • $\begingroup$ The Y axis on this plot doesn't look like count or proportion, what is it? My opinion is that rare values can be kept only if there is a large number if instances, because if the count is lower than 3-5 it's likely to cause more problems than to help. $\endgroup$
    – Erwan
    Nov 14, 2022 at 12:51
  • $\begingroup$ The plot shows the value count for each categorical feature. So "Bulk" is represented approximately 1.2M times, whereas e.g. "Tug" is represented approximately 0.1M times. $\endgroup$
    – fendrbud
    Nov 14, 2022 at 12:57
  • $\begingroup$ Oh ok, sorry I missed the 1e6. So to me this is a case where rare values can be kept or removed, there's no strong reason either way. $\endgroup$
    – Erwan
    Nov 14, 2022 at 15:08


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.