Preserving / fixing class imbalance

Question

Suppose that I have 2 collection $A$ and $B$ of unlabeled animals that are either dogs or cats.

• The dogs in $A$ and the dogs in $B$ are not necessarily identical, other than the fact that they are both dogs. Same for the cats.
• I know for a fact that dogs are much rarer
• I have a variable $Y$, say, owner's youth indicator, that is positively correlated with dogs

I would like to sub-sample the data so that they can be labelled and in turn use labelled data to train a dog/cat classifier.

Question: should I use this variable $Y$ to skew the random sampling so that the dog population in the sub-sampled $A$ and sub-sampled $B$ are likely higher than they are in reality?

Are there references and guidances for this that I can use to think through?

Answer 1

There are many references and guidance for this kind of question. I recommend you consider the metric that matters most. Consider recall precision, and/or F1.

You might consider creating class weights to perform cost-sensitive learning (see here). Basically, this tells the model to place a specific weight on classifying this class incorrectly.

Other suggestions would be to upsample or downsample your minority class. However, this should be done with caution. If the probability of your minority class in production truly is a minority class, this can distort the predicted probabilities, leading to poor classifier performance in production. You should check out this great post. This is probably the best post I have seen on this topic. Read the complete thread.

Answer 2

Using Bayes theorem, we can write the posterior probability of class membership as:

$P(C|x) = \frac{P(x|C)P(C)}{P(x)}$

The posterior probability of class membership is the ideal information we need for a classification task - if we know that accurately, we can find the Bayes optimal decision rule that minimises the error rate.

Now if we balance the dataset with up- or down-sampling the data, or worse still use e.g. SMOTE, then we are throwing away $P(C)$, which is part of the optimal Bayes decision rule. So why on Earth would we want to do that? The "skew" is useful information!

Now if there is a big imbalance, then we often get a classifier where the optimal Bayes classifier (assuming false-positives and false-negatives are equally bad) assigns ALL patterns to the majority class. The first thing to realise is that if that assumption is correct (i.e. accuracy is a useful metric for your problem), then you don't need to worry about it, that is the optimal solution.

However, in a lot of cases, the misclassification costs are not equal, e.g. for a medical screening problem, telling someone they don't have a serious illness when they actually do is a much worse error than telling them they are ill when they are not. We can build these misclassification costs into the decision rule ("cost-sensitive learning"). Often problems with an imbalance have unequal misclassification costs, but for some reason practitioners obsess about the imbalance rather than finding out what the misclassification costs are and building them in to the model.

Essentially the key is to work out what is important for your application, and use that information in choosing the performance metric and in building the decision rule.

Now if you have a very small dataset, then imbalance can sometimes result in an undue bias against the minority class, however when that happens there is very little you can really do to fix it other than gather more data (as the problem is that we don't have enough data from the minority class to properly characterise its distribution).

Another issue is that the class frequencies in the training data are different from those in operational use. However, that is a separate problem, which I address in my answer to a related question here.