2
$\begingroup$

I am currently solving a classification problem for an imbalanced data set (approximately 17% of the minority class). I split the data using a stratified k-fold split from sklearn (Stratified shuffle split) after which I oversample the train data, using ADASYN, and fit the oversampled train data (approximately 250k+ instances after oversampling) to gradient boosting classifier. Oversampling has a huge effect on the performance, recall measure improves from 7% to 75%. Is this possible? If not, any ideas what could be going wrong?

My main question about this is could this improvement be possible?

$\endgroup$
  • $\begingroup$ have a look at this datascience.stackexchange.com/questions/64706/… $\endgroup$ – IamTheRealFord Dec 16 '19 at 8:45
  • $\begingroup$ So you oversample your data, then you split k folds, and perform cross validation ? is that it? just to make sure i understand. $\endgroup$ – Blenz Dec 16 '19 at 9:24
  • $\begingroup$ I first split, then I oversample my train data. $\endgroup$ – 19dr95 Dec 16 '19 at 11:43
1
$\begingroup$

The evaluation of an imbalanced dataset should be done with more than one single metric since you need to evaluate the performance on both the majority and the minority classes.

In the case that is mentioned the performance is evaluated only with recall (the true-positive rate). It could be that the model is now focusing on the oversampled minority class (especially if it's the new majority class) and having a weaker performance on the original majority class, and I assume that the precision or the false-negative rate were decreased.

There are several evaluation metrics that are considering the performance of both classes that are not sensitive to the classes imbalance - F1 score, Geometric mean, and the most common the ROC/PR AUC

| improve this answer | |
$\endgroup$
  • $\begingroup$ +1; by oversampling, you are (primarily) inflating the probability scores, and if you are using the default cutoff of 0.5 this means your model will now predict many more samples to be the positive class, hence your recall will increase. At the same time, your precision will likely drop. Aside from the more-symmetric metrics, consider also performing a cutoff analysis (instead of relying on the default 0.5). $\endgroup$ – Ben Reiniger Dec 23 '19 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.