1
$\begingroup$

I have a dataset with 4712 records. Label 1 is 1558(33%) and Label 0 is 3154 (67%)

a) Currently when I run the model and analysis as is (without sampling techniques), I get an F1-score of 71-77. I chose F1-score and AUC score as the metric as my dataset is imbalanced. (Atleast that's what I felt looking at class proportion). My AUC also ranges between 80-83 for tree based models. Screenshot of models with imbalanced data is given below

enter image description here

b) When I under-sample the majority class, I get all the metrics like F1-score/accuracy/AUC above 80 but less than 85

screenshot below

enter image description here

Now I am not sure which one should I consider? I know it's all about trade-offs.

My objective is to avoid/minimize the misclassifications

Based on your experience in ML projects, what would you guys suggest?

Can someone enlighten me with some reasons why to choose one over the other?

$\endgroup$

1 Answer 1

1
$\begingroup$

Second.

why? you said you objective is to minimize misclassifications which equates to maximizing F1 score, and you achieve that in inbalaced situations when you down-sample the majority class (one approach to maximize)

Potential Pitfall:

It could be that you just got rid of of some important infomration in the majority class and you are over-fitting on what is left. In some cases 1500 datasamples would suffice, but I it could be that it doesn not have enough information. I would rather advise to upsample the smaller class, for example using ADASYN, SMOTe , knnSMOTE etc... Only you can answer this question, by knowing some meta-information about your dataset, quality rules, are you expecting huge covariate-shift, is the dropped data really that different in distribution, etc...

Then do the analysis. If there is one thing you should always ask for is diverse and informative dataset, and more data will (almost) always beat better algorithm.

$\endgroup$
2
  • $\begingroup$ I tried with SMOTE but the result looks more or less the same as option a) where the F1-score is between 72-77. Even after upsampling using SMOTE, I don't see much difference in metric $\endgroup$
    – The Great
    Dec 19, 2019 at 9:39
  • $\begingroup$ Machine learning is SOFT science, with weakish mathematical theory. You do not have definitive answers on all questions, but best guesses. $\endgroup$
    – Noah Weber
    Dec 19, 2019 at 10:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.