# Confusion matrix. "How close I am to the diagonal?". Is there such metric?

I have a question regarding confusion matrices.

To start, we discuss the case of multi-class classification so the confusion matrix has dimension, for example 4 times 4, for classification task with 4 possible outcomes.

The correct predictions should bring all the answers on the diagonal. Is it possible to rate though how close are the results on the diagonal? For example, is much better to predict a class very close to the current one (so guess 3 instead of 4) in comparison to predict wrongly a further away class (predict 4 while it was 1).

Is there a metric that can take that into consideration? Rate kinder classes that are have missed the ideal target by a bit and penalize stronger misclassification that are further away from the real class?

Any suggestions on how to proceed?

• Say the categories in an image recognition problem are dog, cat, and horse. Is it worse to call a dog a cat that it is to call a dog a horse?
– Dave
Sep 3, 2020 at 19:56

The classes that define the columns/rows can be arbitrarily rearranged. Therefore, the "distance" of a misclassification to the diagonal has no meaning. So no, there is no such metric.

I like @Dave's comment: "Is it worse to call a dog a cat than it is to call a dog a horse?"

Maybe you'd ask yourself, "some classes feel closer together than others"

For example, we could create a confusion matrix like this:

$$\begin{matrix} &&&&PREDICTED\\ &&Person & Woman & Man & Camera & TV \\ T&Person & 33 & 5 & 3 & 0 & 1 \\ R&Woman & 10 & 50 & 2 & 22 & 0 \\ U&Man & 12 & 23 & 47 & 1 & 13 \\ T&Camera & 4 & 2 & 7 & 24 & 9 \\ H&TV & 3 & 5 & 8 & 13 & 11 \end{matrix}$$

(Where $$Person$$ denotes non-binary of course).

It feels like misclassifying $$Camera$$'s as $$Woman$$ should be "more wrong" than misclassifying $$Man$$ as $$Woman$$. After all, women aren't objects.

However, in the world of unfeeling classifiers, a "woman" has no meaning, and neither does "object". Hence we call such classifiers "tools".

In situations where there's some notion of distance, you'd use regression rather than classification. You can use regression even for discrete dependent variables. In other words, you need either numeric or ordinal data for distance to make sense.

To use the example in your comment of "classifying" a variable that represents performance:

• Continuous: If you measure performance as a continuous variable, then it's clear to use regression.

• Ordinal: But even if you measure performance on, say, an integer scale from 1-10, you can still regress the data. (As an aside, in practice, all measurements can be considered are discrete if you consider that they are limited by resolution/precision). You can also map ordered concepts to discrete but numeric values. For example, the Likert scale (Strongly Disagree, Disagree, Neutral, Agree, and Strongly Agree) can be mapped to integers 1-5. However, the reason you can't directly determine distance without mapping to numerical values is because there's no intrinsic distance between the nominal values. Strongly Disagree could be two units away from Disagree, and maybe Neutral is a billion units away from Agree.

• Nominal: If you measure performance using words like "good", "decent", "fine" where there's no clear ordering, then distance makes no sense.

• Thanks very nice answer. I am thinking for the case that classes represent performance and you want to measure how good the prediction are. If the prediction is a bit off that is still very good since you get a good feeling on how performance will be like (with bad being predicting a very good performance when it will be very bad in reality). Sep 4, 2020 at 9:25