In multi-class, is the average accuracy of each class in confusion metrics equal to the accuracy calculated from cross validation?

Question

When I calculate accuracy through cross-validation, it gives me a different accuracy than when I calculate through confusion metrics. Why does it give different accuracy? Is overall calculated accuracy through cross-validation or accuracy-matrix not equal to the average accuracy of each class in the confusion matrix?

For example, I obtain accuracy by using cross-validation is 85% and the average accuracy of each class in confusion metrics is 95%. Why the huge difference in both measures?

Answer 1

Yes, this is actually normal.

This is due to some kind of ambiguity in the definition of accuracy for the multiclass case.

• Intuitively accuracy represents the proportion of correct predictions among all the instances.
• Formally it is defined for binary classification as $\frac{TP+TN}{TP+TN+FP+FN}$. It's easy to see that this represents the proportion of correct predictions (TP and TN) among the total number of instances (which includes the error cases FP and FN).

In binary classification the accuracy of the two classes is equal, since switching positive and negative labels only means switching TP with TN and FP with FN. Note that this is not true for precision and recall, which is why standard classification reports show precision and recall for every class even for only two classes.

But in the multiclass case there are two ways to interpret accuracy:

• The intuitive definition as the proportion of correct predictions. In this case we sum all the values in the diagonal of the confusion matrix (correct cases where true class $X_i$ is predicted as $X_i$) and divide by the total number of instances (including error cases where true class $X_i$ predicted as different class $X_j$, where $i\neq j$).
• The formal definition as $\frac{TP+TN}{TP+TN+FP+FN}$. Now we must choose a class as positive and consider all the other classes as negative, and then repeat for every class. This is the standard one vs rest method, which implies that the labels TP,FP,FN,TN are different depending on which class is picked. In particular this means that all the cases which are actual negative and predicted as negative are considered as true negative, even though the predicted class can be different from the actual class. For example if the chosen positive class is $X_1$, any error between $X_2$ and $X_3$ doesn't count as an error.

The first "intuitive" method is actually the standard way to calculate accuracy in the multiclass setting. It gives a single accuracy value across all the classes, whereas the second one gives an accuracy value for every class.

As observed by OP, the first method (used in the cross-validation case) gives a lower value than the second one. This is because "correct cases" are counted strictly as "the actual class and predicted class is identical", whereas the second method doesn't take into account the errors which don't involve the current positive class.