1
$\begingroup$

When I calculate accuracy through cross-validation, it gives me a different accuracy than when I calculate through confusion metrics. Why does it give different accuracy? Is overall calculated accuracy through cross-validation or accuracy-matrix not equal to the average accuracy of each class in the confusion matrix?

For example, I obtain accuracy by using cross-validation is 85% and the average accuracy of each class in confusion metrics is 95%. Why the huge difference in both measures?

this is a confusion matrix

$\endgroup$
7
  • 1
    $\begingroup$ Welcome to DataScienceSE. Normally accuracy is calculated for all the classes together, not class by class. How do you calculate the accuracy by class from the confusion matrix? I suspect that you have an error there. $\endgroup$
    – Erwan
    Commented Dec 2, 2021 at 20:17
  • $\begingroup$ Also using accuracy as a metric for a multiclass classification model is not advised. $\endgroup$
    – spectre
    Commented Dec 3, 2021 at 5:04
  • $\begingroup$ First, I calculated the accuracy of each class separately by using the accuracy formula that is (TP+TN)/(TP+TN+FP+FN). Then I took an average of all the classes' accuracy for final accuracy. Is it the right way to calculate accuracy from the confusion matrix? $\endgroup$
    – Fatima
    Commented Dec 5, 2021 at 11:24
  • $\begingroup$ @Fatima ok you are applying "one vs. rest", this looks correct but normally it's not needed for accuracy because you should obtain exactly the same result for every class, otherwise there's something wrong. If you edit your question to show the confusion matrix and how you calculate accuracy I can have a look. $\endgroup$
    – Erwan
    Commented Dec 6, 2021 at 0:41
  • $\begingroup$ Also just a tip: use @somebody when you reply to a comment for people to know that you replied (you can see new comments automatically because it's your question). $\endgroup$
    – Erwan
    Commented Dec 6, 2021 at 0:41

1 Answer 1

2
$\begingroup$

Yes, this is actually normal.

This is due to some kind of ambiguity in the definition of accuracy for the multiclass case.

  • Intuitively accuracy represents the proportion of correct predictions among all the instances.
  • Formally it is defined for binary classification as $\frac{TP+TN}{TP+TN+FP+FN}$. It's easy to see that this represents the proportion of correct predictions (TP and TN) among the total number of instances (which includes the error cases FP and FN).

In binary classification the accuracy of the two classes is equal, since switching positive and negative labels only means switching TP with TN and FP with FN. Note that this is not true for precision and recall, which is why standard classification reports show precision and recall for every class even for only two classes.

But in the multiclass case there are two ways to interpret accuracy:

  • The intuitive definition as the proportion of correct predictions. In this case we sum all the values in the diagonal of the confusion matrix (correct cases where true class $X_i$ is predicted as $X_i$) and divide by the total number of instances (including error cases where true class $X_i$ predicted as different class $X_j$, where $i\neq j$).
  • The formal definition as $\frac{TP+TN}{TP+TN+FP+FN}$. Now we must choose a class as positive and consider all the other classes as negative, and then repeat for every class. This is the standard one vs rest method, which implies that the labels TP,FP,FN,TN are different depending on which class is picked. In particular this means that all the cases which are actual negative and predicted as negative are considered as true negative, even though the predicted class can be different from the actual class. For example if the chosen positive class is $X_1$, any error between $X_2$ and $X_3$ doesn't count as an error.

The first "intuitive" method is actually the standard way to calculate accuracy in the multiclass setting. It gives a single accuracy value across all the classes, whereas the second one gives an accuracy value for every class.

As observed by OP, the first method (used in the cross-validation case) gives a lower value than the second one. This is because "correct cases" are counted strictly as "the actual class and predicted class is identical", whereas the second method doesn't take into account the errors which don't involve the current positive class.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.