2
$\begingroup$

I have a question regarding confusion matrices.

To start, we discuss the case of multi-class classification so the confusion matrix has dimension, for example 4 times 4, for classification task with 4 possible outcomes.

The correct predictions should bring all the answers on the diagonal. Is it possible to rate though how close are the results on the diagonal? For example, is much better to predict a class very close to the current one (so guess 3 instead of 4) in comparison to predict wrongly a further away class (predict 4 while it was 1).

Is there a metric that can take that into consideration? Rate kinder classes that are have missed the ideal target by a bit and penalize stronger misclassification that are further away from the real class?

I have been looking a lot about this on the internet but no luck so far.

Any suggestions on how to proceed?

Improve this question
$\endgroup$
1
  • $\begingroup$ Say the categories in an image recognition problem are dog, cat, and horse. Is it worse to call a dog a cat that it is to call a dog a horse? $\endgroup$ – Dave Sep 3 '20 at 19:56
3
$\begingroup$

The classes that define the columns/rows can be arbitrarily rearranged. Therefore, the "distance" of a misclassification to the diagonal has no meaning. So no, there is no such metric.

I like @Dave's comment: "Is it worse to call a dog a cat than it is to call a dog a horse?"

Maybe you'd ask yourself, "some classes feel closer together than others"

For example, we could create a confusion matrix like this:

$ \begin{matrix} &&&&PREDICTED\\ &&Person & Woman & Man & Camera & TV \\ T&Person & 33 & 5 & 3 & 0 & 1 \\ R&Woman & 10 & 50 & 2 & 22 & 0 \\ U&Man & 12 & 23 & 47 & 1 & 13 \\ T&Camera & 4 & 2 & 7 & 24 & 9 \\ H&TV & 3 & 5 & 8 & 13 & 11 \end{matrix}$

(Where $Person$ denotes non-binary of course).

It feels like misclassifying $Camera$'s as $Woman$ should be "more wrong" than misclassifying $Man$ as $Woman$. After all, women aren't objects.

However, in the world of unfeeling classifiers, a "woman" has no meaning, and neither does "object". Hence we call such classifiers "tools".

Edit to address your comment:

In situations where there's some notion of distance, you'd use regression rather than classification. You can use regression even for discrete dependent variables. In other words, you need either numeric or ordinal data for distance to make sense.

To use the example in your comment of "classifying" a variable that represents performance:

  • Continuous: If you measure performance as a continuous variable, then it's clear to use regression.

  • Ordinal: But even if you measure performance on, say, an integer scale from 1-10, you can still regress the data. (As an aside, in practice, all measurements can be considered are discrete if you consider that they are limited by resolution/precision). You can also map ordered concepts to discrete but numeric values. For example, the Likert scale (Strongly Disagree, Disagree, Neutral, Agree, and Strongly Agree) can be mapped to integers 1-5. However, the reason you can't directly determine distance without mapping to numerical values is because there's no intrinsic distance between the nominal values. Strongly Disagree could be two units away from Disagree, and maybe Neutral is a billion units away from Agree.

  • Nominal: If you measure performance using words like "good", "decent", "fine" where there's no clear ordering, then distance makes no sense.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks very nice answer. I am thinking for the case that classes represent performance and you want to measure how good the prediction are. If the prediction is a bit off that is still very good since you get a good feeling on how performance will be like (with bad being predicting a very good performance when it will be very bad in reality). $\endgroup$ – Alex P Sep 4 '20 at 9:25
  • $\begingroup$ @AlexP I added an edit section to address your comment $\endgroup$ – Benji Albert Sep 4 '20 at 14:48
  • $\begingroup$ When it comes to regressing ordinal data I guess there are two options. 1. Convert all measurements to discrete so 1,2,3,4..30 or 2. represent each class with the mean of the bin? For example classes 1-5, 5-10, 11-15 should be converted to 3.5,8.5 and 13.5? $\endgroup$ – Alex P Nov 18 '20 at 7:24
0
$\begingroup$

If your classes have a natural order, like exam grades, you can encode then as consecutive integers. That way you can plot a confusion matrix and also give the mean absolute error in your predictions when they are treated as integer values.

Watch out for unequal distances between adjacent classes, which might make a metric like MAE misleading depending on your application. For example, in the case of exam grades, there might be greater distinction between a C and a B than there is between an A and a B, so that it is more forgivable to mislabel a B as an A.

Improve this answer
$\endgroup$
2
  • $\begingroup$ So an A (4) is twice as much as a C (2) on a 0, 1, 2, 3, 4 scale? Ordinal regression exists to tackle this very problem of having a natural order but not a clear way to define addition and multiplication. $\endgroup$ – Dave Oct 7 '20 at 21:35
  • $\begingroup$ As I tried to say in my answer (maybe not clearly enough?), an A is not necessarily going to be worth twice as much as a C. I wasn’t aware of ordinal regression, but that sounds like a way of dealing with the problem the OP is facing. Perhaps you could post an answer. $\endgroup$ – Nicholas James Bailey Oct 9 '20 at 5:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.