0
$\begingroup$

Im doing a 2-class classification project for an imbalanced data set. The imbalance is about 18%/82%. Im noticing a huge improvement in F1-score when I under-sample; from 16% without under-sampling to 49% with under-sampling. Im wondering if this is possible or that I am doing something wrong? Before under-sampling I split the data set in a train- and test set, apply outlier detection and normalisation (normalisation first on train set, then on test using the scales obtained from train set). Does anybody have any idea, if this improvement is too good to be true, what could be going wrong?? Many thanks in advance!

$\endgroup$
1
  • 1
    $\begingroup$ I guess the question isn't exactly a duplicate of your other question datascience.stackexchange.com/q/64910/55122, but the answer is the same: without adjusting the thresholds, resampling methods shift the proportion of samples' predicted classes, so you shouldn't be surprised by large changes to accuracy, recall, precision, F1, ... $\endgroup$ Jan 7 '20 at 15:34
1
$\begingroup$

Improvement in the evaluation metric by rebalancing depends on the type of classifier algorithm. Certain algorithms are very sensitive to rate imbalances, thus adjusting the respective levels of support will change the algorithm's performance.

One way to empirically verify that only under-sampling is driving the improvement is to run an experiment - gradually increase the level of under-sampling and plot the result on the F1 score.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.