Im doing a 2-class classification project for an imbalanced data set. The imbalance is about 18%/82%. Im noticing a huge improvement in F1-score when I under-sample; from 16% without under-sampling to 49% with under-sampling. Im wondering if this is possible or that I am doing something wrong? Before under-sampling I split the data set in a train- and test set, apply outlier detection and normalisation (normalisation first on train set, then on test using the scales obtained from train set). Does anybody have any idea, if this improvement is too good to be true, what could be going wrong?? Many thanks in advance!
1
-
1$\begingroup$ I guess the question isn't exactly a duplicate of your other question datascience.stackexchange.com/q/64910/55122, but the answer is the same: without adjusting the thresholds, resampling methods shift the proportion of samples' predicted classes, so you shouldn't be surprised by large changes to accuracy, recall, precision, F1, ... $\endgroup$– Ben Reiniger ♦Jan 7, 2020 at 15:34
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Improvement in the evaluation metric by rebalancing depends on the type of classifier algorithm. Certain algorithms are very sensitive to rate imbalances, thus adjusting the respective levels of support will change the algorithm's performance.
One way to empirically verify that only under-sampling is driving the improvement is to run an experiment - gradually increase the level of under-sampling and plot the result on the F1 score.
answered Jan 7, 2020 at 15:17