Why is count encoding effective in improving accuracy? [duplicate]

Question

Can someone please explain why/how Count encoding of categorical features improve accuracy in classification when compared to simply label encoding them ?

I found one explanation in kaggle " Rare values tend to have similar counts (with values like 1 or 2), so you can classify rare values together at prediction time. Common values with large counts are unlikely to have the same exact count as other values. So, the common/important values get their own grouping. " which doesn't seem convincing or I don't understand the reasoning .

Can someone please explain why it performs better than label encoding ? Label encoding can also find frequent patterns to correlate with the target variable , right ?

Answer 1

Its a good question,

I would just like to add my points

Lets assume you have dataset with features (patient: id,execercise_duration: int, fav_products: category) target(diabetes: Binary)
Label encoding will just give numbers to every unique category. Lets assume Category A is ice-cream and Category B is juice and Category C is chocolates. Now if Category A is encoded 1 and Category B is encoded 2 and Category C is 3 but you keep the encoded feature as numerical series then it would simply mean Category C > Category B > Category A (since 3> 2 > 1). But is it the right information to send to model ?

I guess not. Intution says people with fav_products as ice-cream and chocoloate will be diabetic. Category A and B and C just represent three different things nothing is large or small in them.

But if you send frequency or count then lets say more observation in data are of ice-cream, chocoloates and less are of juice. Becuase usually icecream and chocolate are more desirable food than juice. Frequency or count of ice-cream and chocolate will be more than juice. So keeping frequency or count encoded feature as numerical can give information to model that when this encoded feature value is high outcome is diabetes and when it is low outcome is non-diabetic.

Note: A more complex model like decison tree may be able to give good accuracy even with label encoding atleast for this simple example.

Answer 2

My 2 cents.

1. Count encoding includes additional information like frequency of occurance (while at the same time disregarding insubstantial differences) which, in general, is more helpful information than the index of a label as in Label encoding.

2. Count encoding can reduce the curse of dimensionality (ie learning in high-dimensional manifolds) which is known to reduce preformance, unlike Label encoding.

An analogy in mathematics is modulo arithmetic; It is known (in fact a theorem) that some (complex) equations do not have solutions if they do not have solutions modulo some numbers. Since modulo arithmetic is faster and easier it in fact reduces several complex problems to simpler ones, by the sole effect of combining numbers into equivalence classes (modulo operation).

Is it a panacea? Of course not, but certainly provides for easier solutions to complex problems in the cases that it holds true.