How can I deal with tiny categories?

Question

I'm playing around with UCI Bank Marketing Dataset. So, there is a categorical variable named default which tells us if client "has credit in default". That variable has three options: no, yes and unknown. Look at the distribution of it:

no         32588
unknown     8597
yes            3

As you can see, we meet yes in only 3 cases and my question is how to deal with such tiny categories in general? Should I just exclude that from the dataset every time I come across it? Or maybe I should make something like oversampling but merely for that cases?

I'm asking because I'm concerned about its impact on a classification task. As far as I understand, if all of these yes will fall into validation or test parts of the dataset during partitioning, it will distort a metric's result.

Answer 1

It totally depends on the target task and the importance of the "tiny class" for this task:

• In some tasks with multiple classes where there is no particular emphasis on any specific class, "tiny classes" can simply be discarded.
• But in the case of this dataset, the natural target task is to detect default cases, and there's little point in an application which classifies customers between "no" or "unknown" default cases. This means that despite its small size, the "yes" class is very important for the most relevant application of this dataset.

There's no obvious answer to the question of how to deal with a class like this:

• Oversampling it is an option, but this would almost certainly introduce some bias in the model.
• My prefered option here would be to consider the "unknown" class as unlabelled data and try to apply some kind of semi-supervised learning.
• In my opinion this kind of imbalance is close to anomaly detection territory. Normally anomaly detection is unsupervised, but maybe there would be something to investigate here.

Answer 2

I second @Erwan's observation that which categories to censor depends on the purpose of the analysis. This dataset is bank credit data collected with a view to a single outcome variable:

21 - y - has the client subscribed a term deposit? (binary: 'yes','no')

I feel comfortable in discarding the 3 out of 40K observations of "yes" for this purpose, and reducing the categories to two will also allow it to be converted into a dummy variable (0/1). If the variable is a factor, don't forget to re-level.

Answer 3

How you handle this depends on the variable's importance in predicting the target variable. A quick test would be to compute the event-rate of target variable across the distinct classes and determine if the distinct classes significantly separate the target variable. If there is no separation, dropping the variable makes more sense, otherwise you consider dropping the entire records with the tiny class or merging together with other classes. Do remember also that oversampling for purposes of model performance metrics are done when the class imbalance is on the target variable and not the input variable.

Answer 4

With only three observations, a model won't be able to make any confident estimate for the contribution. I think your best bet is to use some domain knowledge to encode the column (no=0, unknown=1, yes=1.2? 0.7? ...), or just drop the column if it's not informative. Dropping the rows is fine too, but in production what would you want the model to do if it encountered another "yes"?