When exactly should I use weighted loss?

Question

Do I have to use it in any case when the class distribution is imbalance (Train: class A:10%, B:90% and Test: class A:10%, B:90%)

or when it is different (Train: class A:10%, B:90% and Test: class A:50%, B:50%)

Thanks!

Answer 1

As with almost every problem in real life, regarding how to handle class imbalance there is no golden rule to follow for any case and systematically obtain good results, only some recipes that you can try.

When your class distribution is imbalanced (Train: class A:10%, B:90% and Test: class A:10%, B:90%)), one recipe that can improve your results is using reweighting to give more importance to the minority class. There are, however, other approaches, like oversampling the minority class, or subsampling the majority class. Another option would be to use a loss function that is tolerant to class imbalances.

Having a different distribution between the training and test sets (Train: class A:10%, B:90% and Test: class A:50%, B:50%) can also be handled with the approaches above. However, you should think about why you have such a difference: while the existence of class imbalance in the training and validation data can be caused by the distribution of the data in real life, the differences in class imbalance between training and test data is a consequence of the creation process of those datasets, either caused by a conscious decision or by chance. Here are two examples that could lead to this situation:

1. In this case, the 50%/50% ratio reflects the reality of the data that will be received in the production environment, so we decided to define a test set with that ratio and using the rest of the data for training. It happened that the rest of the data had this imbalance. In this case, you should mitigate the problem of the data imbalance in the training data, maybe with some techniques like reweighting or resampling.

2. In this case, the 90%/10% ratio reflects the reality of the data that will be received in the production environment. However, we created the test set and the data set separately, maybe with less quality control for the training data. This led the team in charge of the training data to make some decisions about the data collection and the team in charge of the test data to take different decisions. The differences in the class imbalance between training and test data and casual, not driven by any conscious decision. In this case, this is a problem in how you have created your datasets. What you have to do in those cases, is to re-split your data to have the same distribution in both sets, trying to have a test set that is as close as possible to the real data and, if possible, having the same distribution in the training and test sets.

In all these cases, nevertheless, we should take into account the function we use to measure how good our model is. If the real data has a strong class imbalance, we should not choose the accuracy but something like the AUC.