Counting the values of a column using pandas I got the following result:

Human                 195
Mutant                 62
God / Eternal          14
Cyborg                 11
Human / Radiation      11
Android                 9
Symbiote                8
Kryptonian              7
Alien                   7
Demon                   6
Atlantean               5
Alpha                   5
Asgardian               5
Cosmic Entity           4
Inhuman                 4
Human / Altered         3
New God                 3
Animal                  3
Saiyan                  2
Eternal                 2
Frost Giant             2
Human-Kree              2
Demi-God                2
Human / Cosmic          2
Vampire                 2
Metahuman               2
Amazon                  2
Icthyo Sapien           1
Czarnian                1
Rodian                  1
Martian                 1
Clone                   1
Zombie                  1
Maiar                   1
Yoda's species          1
Human-Vulcan            1
Zen-Whoberian           1
Mutant / Clone          1
Korugaran               1
Dathomirian Zabrak      1
Parademon               1
Kaiju                   1
Flora Colossus          1
Human-Spartoi           1
Yautja                  1
Ungaran                 1
Human-Vuldarian         1
Neyaphem                1
Xenomorph XX121         1
Bizarro                 1
Human / Clone           1
Gungan                  1
Bolovaxian              1
Talokite                1
Luphomoid               1
Tamaranean              1
Kakarantharaian         1
Spartoi                 1
Strontian               1
Gorilla                 1
Name: Race, dtype: int64

I am new to data science, but I think that all those value appearing only once in the dataset is not going to help the classifier, so is there a good way to handle those values? I was thinking about grouping all those values that appear less than 5 times, or maybe I should remove the lines. By the way, I do not know if it is important to know, but I want to apply the gaussian naive bayes, knn and logistic regression to this dataset. This column is a feature to predict a binary value.


Probably the best thing to do is use domain knowledge to relabel those into the larger categories. You may be able to replace domain knowledge with an imputation method: remove the rare labels, then fill the newly missing data using the other columns. Finally, the quickest sound idea, which you and Brian have both mentioned: just lump them into an "other" category. I wouldn't just drop them, or predicting on future examples outside the surviving categories will be harmed (both in that the model won't understand them, and that your package will have to know how to even pass them to the model).


Assuming those are target categories, the most common strategies are:

  • Drop them. Do not try to predict categories with single observations because the model will overfit.

  • Bundle them to together into a "dustbin" category. Then the model can learn to predict "dustbin" category membership.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.