Strategies to encode categorical variables with many categories

Question

I was going over the Kaggle competitions IEEE,Categorical Feature Encoding Challenge and one of the ways in which categorical variables have been handled is by replacing the variables by the respective count frequencies of the variables in the column. I understand why we need to transform the variables, but can someone please explain why this approach was used. What exactly is the logic behind it?

Secondly, I was wondering if anyone may be willing to share other techniques of dealing with categorical variables (excluding the obvious one hot encoding) or point me some interesting approaches for the same. Thank you!

Answer 1

Generally, the logic of the categorical count transformation lies in the fact that features with similar frequencies tend to behave similarly. Have words in a corpus as an example, common words share little or no real information whereas uncommon words share more information with an algorithm.

Specifically, certain algorithms (tree-based methods) could even yield rules given an unspecified category from an event count. Say, for example, we have an unknown category whose count is 4. The algorithm may give a rule:

If Column Count is < 5 and N is > 3 = X

That will be exactly the same as if an algorithm took a One-Hot encoded column and gave a rule:

If One-Hot-Encoded-Column is > 0 = X

In that case, a tree-based algorithm will make several rules from many categories using the same count column. But how I said at the beginning, algorithms generalize among populations of similar counts so most likely you will find rules like:

If Column Counts is < 10 and N is > 3 = X

Which will often contain different categories that behave similarly. Just check the models and look for parameters/ importance of the column in question to see this for yourself.

Secondly, I was wondering if anyone may be willing to share other techniques of dealing with categorical variables.

Feature hashing became really popular there not so long ago. Hashing has very nice properties and it's a whole topic learned at schools but the main principle is that if you have a category with high cardinality you decide a minimum number of reduced categories (hashes) that all the categories will have to share. if two categories share the same hash or bucket, that is called a hash collision. Feature hashing doesn't deal with hash collisions because according to some authors (I don't have the reference here) may improve accuracy by forcing the algorithm to pick more carefully the features.

There are many ways we can encode these categorical variables as numbers and use them in the algorithm.

1) One Hot Encoding
2) Label Encoding
3) Ordinal Encoding
4) Helmert Encoding
5) Binary Encoding
6) Frequency Encoding
7) Mean Encoding
8) Weight of Evidence Encoding
9) Probability Ratio Encoding
10) Hashing Encoding
11) Backward Difference Encoding
12) Leave One Out Encoding
13) James-Stein Encoding
14) M-estimator Encoding

Find the below cheatsheet

More Info