-1
$\begingroup$

I made different tests on an imbalanced dataset and got these results:

  • Model 1 = train test validation split + Cross Validation(cv=10) --> f1'micro' 0,95

  • Model 2 = train test split + smote method for imbalanced data. No Cross Validation -->f1'micro' 0,97

  • model 3 = train test validation + smote method --> f1'micro' 0,97

  • model 4 = train test + smote --> f1'micro' 0,98.

I used f1 micro as metric. Can I compare these models with f1 micro or should I take another one like f1 macro? or just the accuracy_score?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
2
$\begingroup$

Selecting the correct scoring metric depends on the business problem you are trying to solve. I would research the differences between f1 micro and macro and determine which scoring metric ultimately tracks performance of your task in a more seemly manner. For example: do you just want to maximize f1 score across all samples? Or do you care about the individual f1's of each class? Answering this will help you determine macro vs micro.

As for the core of your question it's difficult to say without a code sample. In a few cases you are using SMOTE - how are you using it? Are you up-sampling before or after you split your data and use cross validation? Up-sampling before CV/splitting your data can lead to data leakage and will artificially inflate your scoring metrics.

My advice:

  1. Pick a single appropriate scoring metric
  2. Ensure you are using SMOTE in a way that doesn't cause data leakage
  3. Perform experiments & pick the model that maximizes your scoring metric
Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you very much. The original plan was to improve the f1 score with different hyperparameter and models. Unfortunately I had not much knowledge before about imbalanced datasets/different f1 scores etc. Now I red that macro f1 should be good when there is imbalanced dataset. I think I should recreate the whole tests. First round no smote and f1 macro, second round with SMOTE and f1 micro. Your advice No.2. I used SMOTE and than did the train test split. So is that general wrong? I read really different stuff in internet about the use of SMOTE. $\endgroup$
    – martin
    Commented Oct 30, 2020 at 23:20
  • 2
    $\begingroup$ The issue with using SMOTE before you split your data is that the up-sampling method creates similar copies of your data which you then split into testing and training. If the same sample of data (that was essentially duplicated) ends up in your training AND testing set then you are testing on information that you are also training on. See this for more details: datascience.stackexchange.com/questions/82073/… $\endgroup$
    – Oliver Foster
    Commented Oct 30, 2020 at 23:24
  • $\begingroup$ Wow they advice SMOTE only on training data. I will run that 2 times. Once with SMOTE on training samples and f1 micro, the other run without SMOTE and f1 macro.Thank you very much for this valuable answer and comment $\endgroup$
    – martin
    Commented Oct 30, 2020 at 23:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.