3
$\begingroup$

one approach I have tried when preprocessing high cardinality categorical features (for example, US City) is to do a value count of all the values in the data, then take the top x most frequently occurring values (with x depending on the frequency distribution), and then create a binary flag feature like 'is_in_top_x_us_cities'. Or creating one-hot features for each of the top x cities.

Can someone explain the relative disadvantages to this approach as opposed to using something like Weight of Evidence binning?

Improve this question
$\endgroup$
2
  • $\begingroup$ There's another option you haven't considered. It's called target encoding and it's frequently used in kaggle competitions. $\endgroup$
    – Victor Ng
    Commented Mar 11, 2020 at 15:35
  • $\begingroup$ Reference to comment of @VictorNg : kaggle.com/ryanholbrook/target-encoding $\endgroup$
    – Francis Laclé
    Commented Dec 9, 2021 at 13:58

1 Answer 1

Reset to default
1
$\begingroup$

The obvious disadvantage is that you lose information.

Probably the best approach is to generate new features based on the city, using domain knowledge: size of the cities, but also relevant geographical/societal/economic features of those cities. Without domain knowledge and additional data, maybe clustering on your data can help to group together similar cities (though now your model will see that same information in the city-groups as well as in the data you used to create the clusters).

Weight of evidence is sort of like clustering similar cities, but according only to your target variable. If you have cities with very few samples, then the WoE scores will suffer from high variability, so you'll probably want to lump them together anyway.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.