0
$\begingroup$

For example, if i have a feature "colour_codes" that has close to 5000 distinct color codes inside it. And the number of samples/rows is 10 million. Then should I ignore the feature "colour_codes" ?

The single categorical variable has a large number of categories but the number of categories is very small compared to the number of rows/samples(5000/10million =0.0005 or 0.05 percent). Butstrong text each of the categories in that categorical variable can have a significant number of samples(2000).

What's the minimum ratio of the number of categories in a categorical variable to the number of samples should be for ignoring the categorical variable entirely?

$\endgroup$

1 Answer 1

1
$\begingroup$

I do not think there will be an if n < x then ignore answer. As most things with ML, it will depend. Do some experimentation. Investigate if the rare categories are important to the problem. Investigate categorical encoding mechanisms besides 1-hot. Try clustering, like hierarchal clustering, to group categories that have a similar effect. I have used random effects models to replace the category with the random effect coefficient where the remaining features are the fixed effects. These techniques may, or may not, provide the model with more predictive power. Also depends on how much time to solve this problem. Perhaps there is lower hanging fruit in feature selection/data prep to deploy a version of the model that meets the problem then come back in v2 and focus on the smaller aspects that may improve the model.

$\endgroup$
2
  • $\begingroup$ n<x?. what does n and x represent. can you elaborate a bit more?. $\endgroup$
    – insomniac
    Jan 25, 2022 at 14:48
  • $\begingroup$ n and x are the numbers you asked for. "minimum ratio of the number of categories in a categorical variable to the number of samples". There will not be a hard rule applicable to all circumstances. $\endgroup$
    – Craig
    Jan 26, 2022 at 11:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.