3
$\begingroup$

I am dealing with multi-class classifiers. My data is unbalanced. Hence, I need to apply sampling techniques before training (undersampling or oversampling). When I apply undersampling, loss and val_loss, as well as acc and val_acc show a good fit. In this case, is it still necessary to oversample the data? What results should I expect?

$\endgroup$
2
2
$\begingroup$

The only case where I would consider resampling data is when there is a requirement to improve recall for a particular class. Thus the goal would be to force the classifier to predict this class more often, even though it usually means decreasing performance in general.

Resampling is an easy method but rarely the optimal one. In general I'd first do an analysis of the errors made by the classifier, possibly consider alternative designs and/or feature engineering.

$\endgroup$
4
  • $\begingroup$ Why calculate a wrong posterior probability instead of altering the classification threshold? $\endgroup$
    – Dave
    Sep 8 at 17:46
  • $\begingroup$ @dave I agree that setting the classification threshold is a good solution for binary classification (assuming a soft classifier which outputs a probability), but if I'm not mistaken it's not possible in the case of multiclass classification. $\endgroup$
    – Erwan
    Sep 8 at 22:43
  • $\begingroup$ The trouble in a multiclass problem is that if you set a high standard of, say, $0.9$ probability of class membership, you might wind up with no class meeting your standard and having to consider that a grey zone. Frank Harrell would see this as a positive, however. // I have no idea how software handles this, though, and assume most standard packages just pick the class with the highest probability. $\endgroup$
    – Dave
    Sep 8 at 22:55
  • $\begingroup$ @Dave Interesting, thanks. Yes, I think most software apply the one-vs-rest strategy which boils down to picking the class with the highest probability. $\endgroup$
    – Erwan
    Sep 10 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.