I've learned that adjusting imbalanced data when training a CNN affects model performance which got me thinking "what about in ML?" so I've done some testing on my own, you can check it out here -> https://haneulkim.medium.com/handling-imbalanced-class-in-machine-learning-classifiers-1b5c528f427f

I've ended up concluding that as we balance our data using random oversampling our precision and accuracy decrease. I want to know why this is the case...

If it requires me to do more research, link to resource would be greatly appreciated. Thanks!


Intuitively the problem of imbalanced data can be understood like this: if a classifier is not really sure how to classify an instance but it knows that most instances belong to class X, then whenever there's a doubt predicting class X is always the best decision. As a consequence, the classifier unavoidably assigns class X too often since all the "unsure" cases end up being labeled as the majority class X.

So first it's important to understand that resampling is not automatically the "cure" for imbalanced data:

  • Resampling doesn't provide the classifier with more information, it just presents it in a different way in order to force the classifier to pay more attention to the minority class.
  • In case the data is easy to separate, the classifier can do a perfectly good job without resampling. This means that whenever possible it's better to improve the features rather than using resampling, because that's what can actually help the classifier doing its job.

This being said, resampling is a useful technique in some cases. Assuming the standard choice of the minority class as the positive class, resampling will only increase recall at the expense of precision: as said above, the difference happens when the classifier cannot easily predict the class for an instance. In such a case it has two choices:

  • Assigning the negative majority class: more likely to be correct (True Negative), a small risk of False Negative error.
  • Assigning the positive minority class: more likely to be incorrect (False Positive), a small chance to be correct (True Positive)

Without resampling the classifier favors the first option, so it has few FP errors but quite a lot of FN errors. Therefore it can have quite high precision but low recall.

With resampling the classes are equal, so the classifier stops favoring the first option. Therefore it makes less FN errors but more FP errrors, which means that it increases recall at the expense of precision.

  • $\begingroup$ Very helpful! Thanks you! $\endgroup$
    – haneulkim
    Mar 16 '21 at 5:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.