I know that in imbalanced classification, the classifier tends to predict all the test labels as larger class label, but if we use class weight in loss function, it would be reasonable to expect the problem to be solved. So why we need some approaches like down sampling or up sampling for imbalanced classification problem?


1 Answer 1


Class weights do help with the imbalance problem ("resolve" seems too much), but upsampling has a certain advantage on it.

If you think about it, downsampling/upsampling the number of samples in each class to balance the dataset is almost exactly the same as using class weights.

For example, say you have a dataset containing 3 samples divided into 2 classes: and you are training with an MSE loss.

You can choose to upsample the number of samples from class B, which will get you the following cost function over a single epoch:

Here is where the small difference comes into effect, if you are training in a batch gradient descend (a single weights update per epoch), the prediction for the 2 identical B1 samples will be the same, so the loss function can be written as:

Which is exactly the same as using a weighted loss function. However if you are using a mini-batch gradient descend (as most model our days), the 2 different samples may appear in different mini-batches, and so the predictions for them at the same epoch won't be the same (because one of them will pass thru a model that was already updated once).

This is a small difference but sometimes it is important. It means that with weighted classes, the effective learning rate varies between mini-batches. In some cases that can make learning unstable. So, when possible, upsampling is the better approach (practically it produces slightly better results).

The problem is that you can't always upsample/downsample without any worry. If we go back to our example but this time we have a dataset of 5 samples, divided into 2 classes:

Downsampling is problematic - which of the 3 A class samples do we ignore?

Upsampling is problematic - which of the 2 B class samples do we duplicate?

It can be solved by randomly choosing which samples to keep/ignore at each epoch, however with very big datasets, this can lead to slower processing... So the weighted classes is still a valid option.

  • $\begingroup$ "It can be solved by randomly choosing which samples to keep/ignore at each epoch". How is that a solution? What does randomness do for us here? $\endgroup$
    – fabiomaia
    Commented Mar 26, 2019 at 15:44
  • $\begingroup$ Hey @mark, you say that up-sampling produces better results in practice. Do you have any references for that, maybe? Any clue why that happens? $\endgroup$ Commented May 18, 2020 at 7:16
  • $\begingroup$ @Mark.F your description does seem to make sense. You described that for upsampling, mini-batch has different effective learning rates. But in the end, you mentioned this is the problem of weighted class. and we should choose upsampling $\endgroup$
    – Justin Lin
    Commented Jul 5, 2021 at 5:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.