3
$\begingroup$

I'm playing around with UCI Bank Marketing Dataset. So, there is a categorical variable named default which tells us if client "has credit in default". That variable has three options: no, yes and unknown. Look at the distribution of it:

no         32588
unknown     8597
yes            3

As you can see, we meet yes in only 3 cases and my question is how to deal with such tiny categories in general? Should I just exclude that from the dataset every time I come across it? Or maybe I should make something like oversampling but merely for that cases?

I'm asking because I'm concerned about its impact on a classification task. As far as I understand, if all of these yes will fall into validation or test parts of the dataset during partitioning, it will distort a metric's result.

$\endgroup$
1
  • 2
    $\begingroup$ Important note: the default variable IS NOT the target of this dataset. $\endgroup$
    – UniqueName
    Oct 19 at 11:22
6
$\begingroup$

It totally depends on the target task and the importance of the "tiny class" for this task:

  • In some tasks with multiple classes where there is no particular emphasis on any specific class, "tiny classes" can simply be discarded.
  • But in the case of this dataset, the natural target task is to detect default cases, and there's little point in an application which classifies customers between "no" or "unknown" default cases. This means that despite its small size, the "yes" class is very important for the most relevant application of this dataset.

There's no obvious answer to the question of how to deal with a class like this:

  • Oversampling it is an option, but this would almost certainly introduce some bias in the model.
  • My prefered option here would be to consider the "unknown" class as unlabelled data and try to apply some kind of semi-supervised learning.
  • In my opinion this kind of imbalance is close to anomaly detection territory. Normally anomaly detection is unsupervised, but maybe there would be something to investigate here.
$\endgroup$
3
  • 1
    $\begingroup$ Erwan, thank you for answering. I'm sorry, but it seems like I wasn't clear enough in my explanations. The "default" variable of this dataset is not the target, it's just one of many properties of each case. The whole dataset is about marketing campaigns which were based on phone calls and "the classification goal is to predict if the client will subscribe (yes/no) a term deposit (variable y)". $\endgroup$
    – UniqueName
    Oct 19 at 11:19
  • 2
    $\begingroup$ @UniqueName Oh ok, well it's a good illustration of my first point then: for this task, you really don't care about the 3 "yes" cases, so you could simply discard them. My guess is that you could probably discard the whole feature, since it's unlikely that the distinction "no" vs. "maybe" would help the model. $\endgroup$
    – Erwan
    Oct 19 at 15:11
  • $\begingroup$ Got it. Thank you so much! $\endgroup$
    – UniqueName
    Oct 19 at 15:15
1
$\begingroup$

I second @Erwan's observation that which categories to censor depends on the purpose of the analysis. This dataset is bank credit data collected with a view to a single outcome variable:

21 - y - has the client subscribed a term deposit? (binary: 'yes','no')

I feel comfortable in discarding the 3 out of 40K observations of "yes" for this purpose, and reducing the categories to two will also allow it to be converted into a dummy variable (0/1). If the variable is a factor, don't forget to re-level.

$\endgroup$
1
  • 1
    $\begingroup$ Yep, thank you, it's a nice idea! P. S. Just for future reference for the guys, who, as well as me, don't know what is factor and re-level terms which were mentioned by Richard: it's from R-language and you can grasp more info here: towardsdatascience.com/… $\endgroup$
    – UniqueName
    Oct 20 at 10:49
1
$\begingroup$

How you handle this depends on the variable's importance in predicting the target variable. A quick test would be to compute the event-rate of target variable across the distinct classes and determine if the distinct classes significantly separate the target variable. If there is no separation, dropping the variable makes more sense, otherwise you consider dropping the entire records with the tiny class or merging together with other classes. Do remember also that oversampling for purposes of model performance metrics are done when the class imbalance is on the target variable and not the input variable.

$\endgroup$
1
  • $\begingroup$ Thank you for another great point! What is good about this approach that it is based not just on our personal analysis or field knowledge, but also on the variable's real impact on the target. $\endgroup$
    – UniqueName
    Oct 20 at 11:02
1
$\begingroup$

With only three observations, a model won't be able to make any confident estimate for the contribution. I think your best bet is to use some domain knowledge to encode the column (no=0, unknown=1, yes=1.2? 0.7? ...), or just drop the column if it's not informative. Dropping the rows is fine too, but in production what would you want the model to do if it encountered another "yes"?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.