0
$\begingroup$

The method I saw that's generally used to deal with large # of nominal variables is to keep the most frequent variables and introduce a new "other" category. But that's not possible with my data with equal # of tuples for each nominal variable. How do I handle this? I have 70 tuples for each of the 250 categories.

Edit: After researching more, I think I should use target encoding with prior smoothing but could someone suggest an appropriate value for the smoothing parameter in python?

Improve this question
$\endgroup$
1
  • $\begingroup$ is "tuples"=observations, and "nominal variable"=category inside a single feature? $\endgroup$
    – Ben Reiniger ♦
    Commented Jan 3, 2023 at 1:32

1 Answer 1

Reset to default
0
$\begingroup$

I think this depends on the size of your dataset: if every level (possible value of the feature) occurs frequently enough, one-hot encoding is still probably fine, especially if your model can accept sparse data structures.

If the levels are fairly rare in absolute numbers, then I agree that (some flavor of) target encoding, especially with smoothing, is a good approach. Ideally, treat the smoothing parameter as a hyperparameter of the modeling pipeline to tune. The default is currently 1, but will be increased to 10 in a near-future version.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.