1
$\begingroup$

I am building a CNN model for pitch estimation using a song recording. Pitch estimation is done by inputting spectrogram to CNN model and make the CNN predict pitch sequence (250 pitch values per recording) from that spectrogram. For the evaluation metrics, I am using Accuracy and F1 Score. Sample of overall test result are given below using mean measurement. CNN Sample test result

Some notes:

  1. Val-Acc is the validation accuracy. I am using this to see how well the model analyze new data that is not given during training.
  2. Delta acc is the difference value between accuracy and val-acc.

Right now, I am wondering how can I explain the relationship between Accuracy and F1 Score. My supervisor said to me that accuracy is measured to get how accurate the model performs, and F1 is how well the model performs. Is the relationship really like that? May I get some insight on how to explain the relationship between them?

$\endgroup$

1 Answer 1

1
$\begingroup$

Saying that accuracy is measured to get how accurate the model performs, and F1 is how well the model performs

This doesn't mean anything, it's obviously too vague.

The first things to check in order to understand this relationship are the definitions of accuracy and F1-score. Wikipedia has a good page which explains how different classification evaluation measures are related.

Observations on your results:

  • The accuracy and F1-score are almost identical everywhere. This suggests that your data is probably quite well balanced, i.e. the difference in the number of positive vs. negative instances isn't very big. Why? Because if the data was imbalanced then the model would over-predict the majority class, and this would cause the F1-score to be much lower than the accuracy: assuming the majority class is the negative class, the recall would be somewhat low but the accuracy could still be high because most instances (majority class) would be correctly predicted.
  • As a consequence, there's no insight to gain from analyzing the relationship between accuracy and F1-score since they're virtually identical. The small differences might be due to the geometrical mean between precision and recall. F1-score is more informative in case of imbalanced data, but this is not the case here.
  • The F1-score is calculated only on the training data. It would be more useful to calculate it on the validation data.
  • There's some serious overfitting happening especially with the high learning rates, but with the low learning rates the fact that difference between training and validation accuracy increases is also worrying. Maybe the model is too complex or there are not enough instances in the data. Ideally the two accuracy values should converge.
$\endgroup$
4
  • $\begingroup$ Thank you for the explanation. So, should I use F1 to evaluate my model, or accuracy is just enough? My model has 51 class of pitch, data distribution is quite even. Also, for the overfitting problem, I have found the solution not to use high learning rates. The high learning rates are not used, they are presented just for experiment only to find a good learning rate. $\endgroup$ Commented Apr 30, 2021 at 15:56
  • $\begingroup$ @DionisiusPratama I think you can use one or the other but I always advise to use F1-score simply because it's more robust, it works whether the data is balanced or not whereas accuracy can be misleading in some cases. about overfitting: according to your table the low learning rates also cause a lot of overfitting, normally the difference between training and validation performance shouldn't be that high. $\endgroup$
    – Erwan
    Commented Apr 30, 2021 at 20:36
  • $\begingroup$ Okay, I understand. Thank you for your suggestion! Now I am more curious on the overfitting topic. At how many percent we can say a model is overfitting a lot? Is 5%-10% can be considered a lot? How to consider and define the "lines" between lots and few? Anyway, on my conclusion, I use 0.00001 learning rate value. $\endgroup$ Commented May 1, 2021 at 5:55
  • 1
    $\begingroup$ It's impossible to define a particular limit since it depends so much on the data, but I would say that as soon as the difference is more than say 2-3% there is a significant chance that the model is overfit. Overfitting is when the model focuses too much on details in the training data instead of generalizing so it's a matter of balance between (1) the size and diversity in the training set and (2) the complexity of the model (typically number of parameters). so one needs to increase (1) or decrease (2) (or both) in order to get rid of overfitting. $\endgroup$
    – Erwan
    Commented May 1, 2021 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.