How to fix class imbalance in training sample?

Question

I was very recently asked in a job interview about solutions to fix an imbalance of classes in the training dataset. Let's focus on a binary classification case.

I offered two solutions: oversampling the minority class by feeding the classifier balanced batches of data, or partitioning the abundant class such as to train many classifiers with a balanced training set, a unique subset of the abundant and the same set of the minority. The interviewers noded, but I was later cut off and one of the knowledge gaps they mentioned was this answer. I know now that I could have discussed changed the metric..

But the question that pops in my mind now is: is it really a problem to train a classifier with 80% class A if the testing set will have the same proportion? The rule of thumb of machine learning seems to be that the training set needs to be as similar as possible to the testing for best prediction performance.

Isn't just in the cases with have no idea (no prior) about the distribution of the testing that we need to balance the classes? Maybe I should have raised this point in the interview..

Answer 1

Actually what they mentioned is right. The idea of oversampling is right and is one of, in general, Resampling methods to cope with such problem. Resampling can be done through oversampling the minorities or undersampling the majorities. You may have a look at SMOTE algorithm as a well-stablished method of resampling.

But about your main question: No it's not only about the consistency of distributions between test and train set. It is a bit more.

As you mentioned about metrics, Just imagine accuracy score. If I have a binary classification problem with 2 classes, one 90% of the population and the other class 10%, then with no need of Machine Learning I can say my prediction is always the majority class and I have 90% accuracy! So it just does not work regardless of the consistency between train-test distributions. In such cases you may pay more attention to Precision and Recall. Usually you would like to have a classifier which minimizes the mean (usually harmonic mean) of Precision and Recall i.e. the error rate is where FP and FN are fairly small and close to each other.

Harmonic mean is used instead of arithmetic mean because it supports the condition that those errors are as equal as possible. For instance if Precision is $1$ and Recall is $0$ the arithmetic mean is $0.5$ which is not illustrating the reality inside the results. But harmonic mean is $0$ which says however one of the metrics is good the other one is super bad so in general the result is not good.

But there are situations in practice in which you DO NOT want to keep the errors equal. Why? See the example bellow:

An Additional Point

This is not exactly about your question but may help understanding.

In practice you may sacrifice an error to optimize the other one. For instance diagnosis of HIV might be a case (I am just mocking-up an example). It is highly imbalanced classification as, of course, the number of people with no HIV is dramatically higher than the ones who are carrier. Now let's look at errors:

False Positive: Person does not have HIV but test says they have.

False Negative: Person does have HIV but test says they don't.

If we assume that wrongly telling someone that he got HIV simply leads to another test, we may take much care about not wrongly say someone he is not a carrier as it may result in propagating the virus. Here your algorithm should be sensitive on False Negative and punishes it much more than False Positive i.e. according to the figure above, you may end up with higher rate of False Positive.

Same happens when you want to automatically recognize people faces with a camera to let them enter in an ultra-security site. You don't mind if the door is not opened once for someone who has permission (False Negative) but I'm sure you don't want to let a stranger in! (False Positive)

Hope it helped.

Answer 2

To your question "is it really a problem to train a classifier with 80% class A if the testing set will have the same proportion?", the answer is "it depends on the cost of misclassifications", but normally it is certainly a problem, because your classifier will be biased to classify an individual into the over-represented class regardless of the values of its features because, in average, doing so increased the chances of correct classification during training.

There are different oversampling and undersampling strategies. Maybe the most popular one is SMOTE (Synthetic Minority Over-sampling Technique), which creates synthetic training data based on k-nearest neighbors.