one approach I have tried when preprocessing high cardinality categorical features (for example, US City) is to do a value count of all the values in the data, then take the top x most frequently occurring values (with x depending on the frequency distribution), and then create a binary flag feature like 'is_in_top_x_us_cities'. Or creating one-hot features for each of the top x cities.

Can someone explain the relative disadvantages to this approach as opposed to using something like Weight of Evidence binning?

  • $\begingroup$ There's another option you haven't considered. It's called target encoding and it's frequently used in kaggle competitions. $\endgroup$ – Victor Ng Mar 11 '20 at 15:35

The obvious disadvantage is that you lose information.

Probably the best approach is to generate new features based on the city, using domain knowledge: size of the cities, but also relevant geographical/societal/economic features of those cities. Without domain knowledge and additional data, maybe clustering on your data can help to group together similar cities (though now your model will see that same information in the city-groups as well as in the data you used to create the clusters).

Weight of evidence is sort of like clustering similar cities, but according only to your target variable. If you have cities with very few samples, then the WoE scores will suffer from high variability, so you'll probably want to lump them together anyway.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.