1
$\begingroup$

I'm trying to train VGG-16 on the Pascal VOC 2012 dataset, which has images with 20 labels (and a given image can have multiple classes present). The examples are highly imbalanced, so I've "balanced" them such that each label is represented roughly equal in the training set.

But this means that for each label, 5% of the total images are positive examples and 95% are negative samples. There is no way to achieve a 50/50 split for all classes.

I'm using binary cross entropy loss and a sigmoid activation at the final VGG layer, since this is a multi-label problem. Binary accuracy looks great but in fact, the results for any given class are pretty dismal (~15% recall). The classifier is not fitting to positive examples and is biased toward reporting a negative result because that matches the data distribution (very few positive samples).

What is typically done in this scenario? The original paper appears to train on mutually-independent classes. Should I be using a custom loss function?

$\endgroup$
2
  • $\begingroup$ Focal loss ended up helping. But note that the SigmoidFocalCrossEntropy loss in Tensorflow-Addons did not behave as expected. I ended up using an implementation I found on Github instead. $\endgroup$
    – trzy
    Jul 25, 2020 at 22:01
  • $\begingroup$ Are you sure the imbalance is a problem in need of fixing? $\endgroup$
    – Dave
    Feb 2 at 12:43

2 Answers 2

1
$\begingroup$

There are other loss functions that are more useful for imbalanced multi-label datasets. One example is from FAIR’s Focal Loss for Dense Object Detection paper. Focal loss function is dynamic based on the predicted probability of each object. The goal is to decreases the dominance of over-represented classes in the total loss term.

$\endgroup$
1
$\begingroup$

Did you try weighted binary cross entropy?

In this method, you assign different weights to the positive and negative samples during training. You can assign higher weights to the positive samples to make the model more sensitive to them. The idea is to penalize misclassifications of the positive class more heavily.

You could even try ensemble methods.

Training multiple models and combining their predictions can often lead to better performance. You can train multiple models using different random seeds or with different configurations and then average their predictions or use more sophisticated ensemble techniques.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.