2
$\begingroup$

I am currently solving a classification problem for an imbalanced data set (approximately 17% of the minority class). I split the data using a stratified k-fold split from sklearn (Stratified shuffle split) after which I oversample the train data, using ADASYN, and fit the oversampled train data (approximately 250k+ instances after oversampling) to gradient boosting classifier. Oversampling has a huge effect on the performance, recall measure improves from 7% to 75%. Is this possible? If not, any ideas what could be going wrong?

My main question about this is could this improvement be possible?

Improve this question
$\endgroup$
3
  • $\begingroup$ have a look at this datascience.stackexchange.com/questions/64706/… $\endgroup$
    – IamTheRealFord
    Dec 16, 2019 at 8:45
  • $\begingroup$ So you oversample your data, then you split k folds, and perform cross validation ? is that it? just to make sure i understand. $\endgroup$
    – Blenz
    Dec 16, 2019 at 9:24
  • $\begingroup$ I first split, then I oversample my train data. $\endgroup$
    – 19dr95
    Dec 16, 2019 at 11:43

1 Answer 1

Reset to default
1
$\begingroup$

The evaluation of an imbalanced dataset should be done with more than one single metric since you need to evaluate the performance on both the majority and the minority classes.

In the case that is mentioned the performance is evaluated only with recall (the true-positive rate). It could be that the model is now focusing on the oversampled minority class (especially if it's the new majority class) and having a weaker performance on the original majority class, and I assume that the precision or the false-negative rate were decreased.

There are several evaluation metrics that are considering the performance of both classes that are not sensitive to the classes imbalance - F1 score, Geometric mean, and the most common the ROC/PR AUC

Improve this answer
$\endgroup$
1
  • $\begingroup$ +1; by oversampling, you are (primarily) inflating the probability scores, and if you are using the default cutoff of 0.5 this means your model will now predict many more samples to be the positive class, hence your recall will increase. At the same time, your precision will likely drop. Aside from the more-symmetric metrics, consider also performing a cutoff analysis (instead of relying on the default 0.5). $\endgroup$
    – Ben Reiniger ♦
    Dec 23, 2019 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.