In sklearn's classification report, is f1 the best accuracy measure?

Question

In the classification_report provided by sklearn, which score should I look at to make the best determination of the accuracy of my model?

             precision    recall  f1-score   support

          0       0.70      0.68      0.69      5007
          1       0.65      0.54      0.59      2270
          2       0.37      0.22      0.28       614
          3       0.74      0.30      0.42       252
          4       0.59      0.42      0.49       262
          5       0.35      0.11      0.17       455
          6       0.34      0.23      0.27       248
          7       0.09      0.05      0.06       133
          8       0.38      0.15      0.21       395
          9       0.43      0.31      0.36       182
         10       0.23      0.12      0.16       230

avg / total       0.60      0.52      0.55     10048

As far as I understand it:

• Precision tells us the amount of samples the classifier has correctly marked as true positive out of all positive results.

• Recall tells us about the number of samples the classifier was able to get correct out of all samples in the set.

• F1-score is the harmonic mean of precision and recall.

Maybe I'm misinterpreting the classification_report, but wouldn't the f1-score give the best view on the performance of the classifier?

Answer 1

You got recall incorrectly. It does not mean "correct out of all the samples". $\hspace{50mm}$

Looking at the image,

$\hspace{50mm}$ Recall = $\frac{TP}{TP+FN}$

It is explained here.

From the above documentation, you can also see that you can modify your F-beta score to suite which to weigh more, precision or recall. Therefore, f-score would give you an overall of how good your classifier is, but if you need to know at which samples your classifier fails, you need precision and recall.

Answer 2

F1 is only useful if Precision and Recall are similar.

That's why SKLearn has micro and macro (weighted) F1.

In Data Science interviews, answering F1 is the most important measure is a sign of a beginner. I tend to agree ... there are so many things to watch out. E.g.

• Accuracy kFold cross-validation
• Standard Deviation high or low? A high level is a sign of a problem (e.g. >5%)
• What does the Learning Curve tell you? Do you have enough data?
• A Recall of 0.70 might be great if you have 0.01% label 1
• With a highly unbalanced dataset, AUROC is important.
• I prefer a slightly lower F1 with a low variance algorithm (e.g. Logistic Regression)
• A high F1 score might not be that great if you overfit a Decision Tree
• ....

good luck!