Why doesn't class weight resolve the imbalanced classification problem?

Question

I know that in imbalanced classification, the classifier tends to predict all the test labels as larger class label, but if we use class weight in loss function, it would be reasonable to expect the problem to be solved. So why we need some approaches like down sampling or up sampling for imbalanced classification problem?

Answer 1

Class weights do help with the imbalance problem ("resolve" seems too much), but upsampling has a certain advantage on it.

If you think about it, downsampling/upsampling the number of samples in each class to balance the dataset is almost exactly the same as using class weights.

For example, say you have a dataset containing 3 samples divided into 2 classes: and you are training with an MSE loss.

You can choose to upsample the number of samples from class B, which will get you the following cost function over a single epoch:

Here is where the small difference comes into effect, if you are training in a batch gradient descend (a single weights update per epoch), the prediction for the 2 identical B1 samples will be the same, so the loss function can be written as:

Which is exactly the same as using a weighted loss function. However if you are using a mini-batch gradient descend (as most model our days), the 2 different samples may appear in different mini-batches, and so the predictions for them at the same epoch won't be the same (because one of them will pass thru a model that was already updated once).

This is a small difference but sometimes it is important. It means that with weighted classes, the effective learning rate varies between mini-batches. In some cases that can make learning unstable. So, when possible, upsampling is the better approach (practically it produces slightly better results).

The problem is that you can't always upsample/downsample without any worry. If we go back to our example but this time we have a dataset of 5 samples, divided into 2 classes:

Downsampling is problematic - which of the 3 A class samples do we ignore?

Upsampling is problematic - which of the 2 B class samples do we duplicate?

It can be solved by randomly choosing which samples to keep/ignore at each epoch, however with very big datasets, this can lead to slower processing... So the weighted classes is still a valid option.