On Google's ML crash course it states:

Good feature values should appear more than 5 or so times in a data set. Doing so enables a model to learn how this feature value relates to the label. That is, having many examples with the same discrete value gives the model a chance to see the feature in different settings, and in turn, determine when it's a good predictor for the label.

This make sense, but if that happens what should we do? For example, consider a dataset with street_name as feature and out 2000 rows, only 4 of them have street_name equal to "Learning St.".

Should we remove those rows containing rare feature values? Or what?

Any insights would be greatly appreciated.


Working with Categorical Features can raise a few challenges. For instance, you might encounter features with high cardinality in certain values or the other case (your case), features with rare categories.

The first thing you can consider is, what you will lose if you drop those rows. What extra info do you get with keeping the rare categories. Here you probably need the domain knowledge of the project to identify the impact of the specific data. If your dataset is small or you don't feel safe to remove those rows, then you should look for alternatives.

The second approach is to group all the rare categories in one new called "Rare" or "Other". So, whenever you have a record with that value, you replace it with the new one.

Finally, you can try either a Dimentionality Reduction such as PCA or a Feature Hashing where you will map multiple categories in one new value.

  • $\begingroup$ But isn't PCA for the whole training data rather than a single feature data? $\endgroup$
    – chikitin
    Sep 23 '19 at 12:52
  • 1
    $\begingroup$ You can transform your categorical column with one-hot encoding and then perform PCA on those columns only. As for the second, just replace all the categories with lower than a limit cardinality with a new value such as "other". $\endgroup$
    – Tasos
    Sep 23 '19 at 12:54
  • $\begingroup$ Thank you. I guess I need into consideration all four approaches. My other question is that how do you implement your second approach? Do you add a boolean feature vector say, rare of length equal to number of feature and whenever an example, if for any feature has low cardinality for any value, we put 1, say (0,1,1,0,0) meaning there are rare values for 2nd an 3rd features. $\endgroup$
    – chikitin
    Sep 23 '19 at 12:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.