2
$\begingroup$

I have a project that I'm working on. The dataset contains many categorical variables and some of them have too many levels (+100). My question is : is there any advice to know the "adequate" number of levels of a variable ? is it based on the number of levels of other variables ? (for example most variables have between 10 and 30 levels and one or two variables have 80 100 levels). For the variables that contain too many levels, I want to take 80% of most frequent levels and put the 20% into a new level "others" but I don't know at which number of levels I should stop (for example : var 1 : 70 levels, var 2 : 100 level, var 3 : 13, var 4 : 30, var 5 : 60, should I apply the 80-20 method starting from 60? 70? 100?) I don't know if I'm being clear but I hope you understand

$\endgroup$
2
$\begingroup$

No, there's no "adequate" number of levels for a categorical variable.

The choice to simplify the data by discarding some levels (for example by using a default category, as you propose) depends on what the goal is (and also number of instances, etc.). Very often this choice is made experimentally, that is by trying different methods (e.g. different thresholds) and observing which one gives the best performance: here you could do a program which tries the different proportions as threshold, then train and test the resulting model for every value, and finally plot the performance for every value.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.