0
$\begingroup$

I am trying to do experiments on multiple data sets. Some are more imbalanced than others. Now, in order to assure fair reporting, we compute F1-Score on test data. In most machine learning models, we train and validate the model via accuracy measure metric. However, this time, I decided to train and validate the model on an F1-score metric measure. Technically, there should be no problems, in my opinion. However, I am wondering if this is the correct approach to go.

Second, when I use this method (training, validation on F1-score), I receive a higher loss error and a lower F1-score on training data than on validation data. I’m not sure why.

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
-1
$\begingroup$

In case of imbalanced class, Accuracy should never be used. As a model can just say every observation belongs to majority class and still gets very high accuracy.

For evaluating imbalanced class we use Precision, Recall or F1-score based on whatever metrics align best with out business objective. If you are getting low F1-Score it means that model is not able to perform well. To tackle the low F1-score please try following :

Sampling : Try to balance the data by using UpSampling or downsampling techniques
Class Weights : Use class weights parameter in ML algorithm to assign weights to classes
Data Augmentation : Use data augmentation techniques like SMOTE to generate more examples
Improve this answer
$\endgroup$
3
  • $\begingroup$ What's the problem with getting high accuracy when the classes are imbalanced? $\endgroup$
    – Dave
    Commented Jan 31, 2022 at 15:09
  • $\begingroup$ Because suppose you are predicting credit card default, 99% do not do fraud and only 1% do fraud. Now in this if model says that no one does fraud it still have a 99% accuracy, though model is very bad $\endgroup$
    – Ashwiniku918
    Commented Jan 31, 2022 at 15:25
  • $\begingroup$ You get $99\%$ accuracy. "Rats! We're only at our baseline." What's the problem? Setting aside the issues with accuracy outlined in the link I posted, this only seems like a problem if you give the number without context. $\endgroup$
    – Dave
    Commented Jan 31, 2022 at 15:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.