Balanced Train set to predict Imbalanced Prediction set

Question

One of the methods to address a classification predictive analysis on an imbalanced set consist on undersample the majority class (others approaches consist on: undersample the majority class, synthesize new minority classes...).

So assume hereafter we use any of those solutions and then we train an algorithm with the new generated data set. Will this trained algorithm be useful to predict further data from this system which is in general imbalanced?

Or to make it more concrete, is it possible in general to train a model with a balanced training set so that we can effectively predict an imbalanced prediction set? Or both should be generarly either balanced or imbalanced?

Thanks!

Sergi

Answer 1

It's possible. Consider KNN. When you train a model, it essentially "remembers" the training set. Now, when you use the model to predict an unbalanced data set, the model simply read off from the memory, so there's no problem.

For example, imagine you train a model with an uniformly distributed age group from 10 to 80. Now you have a test set where everybody is 70-80 years old. KNN simply find out the neighbors for your old people (say 60-80 old people in your training data) and then compute a weighted average.

Using a balance training set to predict a imbalanced test set is not super challenging. The other way around is much more challenging.

Answer 2

Class balancing is necessary when the loss function you are minimising during training is not the same as the metric you are using for evaluation. The answer to the question:

is it possible in general to train a model with a balanced training set so that we can effectively predict an imbalanced prediction set?

depends on the choice of loss function and measure of "effectiveness" (the evaluation metric). If both classes are given equal importance in evaluation (eg. ROC AUC), and not in the loss function, then balancing will lead to increased performance.

Answer 3

The answer to this question is very related to the practical problem you are working with. I spent three years modifying SMOTE algorithm to use it with one problem and the next two years trying to design a biased classifier to deal with another problem. In fact, in academia we care a lot about average accuracy comparing to overall accuracy. Here it comes the point that you should deal with the imbalance data problem. But in practical, in many cases you will find that they care about overall accuracy and not average accuracy. When the cost of missing any of the samples is the same then it makes since. If you do not have any consideration,then the question is: can I maximize both average and overall accuracy? This depends on the features you are using. In some cases, they cost of missing the minority samples is higher, then you need to design a biased classifier. The answer to your question: If balancing the data can maximize both of average and overall accuracy, it is better to do it. If balancing the data can not maximize both of them , then you need to think about the problem requirements