0
$\begingroup$

In the classification_report provided by sklearn, which score should I look at to make the best determination of the accuracy of my model?

             precision    recall  f1-score   support

          0       0.70      0.68      0.69      5007
          1       0.65      0.54      0.59      2270
          2       0.37      0.22      0.28       614
          3       0.74      0.30      0.42       252
          4       0.59      0.42      0.49       262
          5       0.35      0.11      0.17       455
          6       0.34      0.23      0.27       248
          7       0.09      0.05      0.06       133
          8       0.38      0.15      0.21       395
          9       0.43      0.31      0.36       182
         10       0.23      0.12      0.16       230

avg / total       0.60      0.52      0.55     10048

As far as I understand it:

  • Precision tells us the amount of samples the classifier has correctly marked as true positive out of all positive results.

  • Recall tells us about the number of samples the classifier was able to get correct out of all samples in the set.

  • F1-score is the harmonic mean of precision and recall.

Maybe I'm misinterpreting the classification_report, but wouldn't the f1-score give the best view on the performance of the classifier?

$\endgroup$
3
$\begingroup$

You got recall incorrectly. It does not mean "correct out of all the samples". $\hspace{50mm}$

Looking at the image,

$\hspace{50mm}$ Recall = $\frac{TP}{TP+FN}$

It is explained here.

From the above documentation, you can also see that you can modify your F-beta score to suite which to weigh more, precision or recall. Therefore, f-score would give you an overall of how good your classifier is, but if you need to know at which samples your classifier fails, you need precision and recall.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Answer might be improved, if it helped correct OP's misunderstanding regarding "best" metric. The metric to select a model on is the one that most closely matches the goal and intended use for a model. As you suggest, you can also look at multiple metrics if you are interested in error analysis. $\endgroup$ – Neil Slater Oct 31 '17 at 10:01
0
$\begingroup$

F1 is only useful if Precision and Recall are similar.

That's why SKLearn has micro and macro (weighted) F1.

In Data Science interviews, answering F1 is the most important measure is a sign of a beginner. I tend to agree ... there are so many things to watch out. E.g.

  • Accuracy kFold cross-validation
  • Standard Deviation high or low? A high level is a sign of a problem (e.g. >5%)
  • What does the Learning Curve tell you? Do you have enough data?
  • A Recall of 0.70 might be great if you have 0.01% label 1
  • With a highly unbalanced dataset, AUROC is important.
  • I prefer a slightly lower F1 with a low variance algorithm (e.g. Logistic Regression)
  • A high F1 score might not be that great if you overfit a Decision Tree
  • ....

good luck!

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.