The classes that define the columns/rows can be arbitrarily rearranged. Therefore, the "distance" of a misclassification to the diagonal has no meaning. So no, there is no such metric.
I like @Dave's comment: "Is it worse to call a dog a cat than it is to call a dog a horse?"
Maybe you'd ask yourself, "some classes feel closer together than others"
For example, we could create a confusion matrix like this:
$ \begin{matrix}
&&&&PREDICTED\\
&&Person & Woman & Man & Camera & TV
\\
T&Person & 33 & 5 & 3 & 0 & 1
\\
R&Woman & 10 & 50 & 2 & 22 & 0
\\
U&Man & 12 & 23 & 47 & 1 & 13
\\
T&Camera & 4 & 2 & 7 & 24 & 9
\\
H&TV & 3 & 5 & 8 & 13 & 11
\end{matrix}$
(Where $Person$ denotes non-binary of course).
It feels like misclassifying $Camera$'s as $Woman$ should be "more wrong" than misclassifying $Man$ as $Woman$. After all, women aren't objects.
However, in the world of unfeeling classifiers, a "woman" has no meaning, and neither does "object". Hence we call such classifiers "tools".
Edit to address your comment:
In situations where there's some notion of distance, you'd use regression rather than classification. You can use regression even for discrete dependent variables. In other words, you need either numeric or ordinal data for distance to make sense.
To use the example in your comment of "classifying" a variable that represents performance:
Continuous: If you measure performance as a continuous variable, then it's clear to use regression.
Ordinal: But even if you measure performance on, say, an integer scale from 1-10, you can still regress the data. (As an aside, in practice, all measurements can be considered are discrete if you consider that they are limited by resolution/precision). You can also map ordered concepts to discrete but numeric values. For example, the Likert scale (Strongly Disagree
, Disagree
, Neutral
, Agree
, and Strongly Agree
) can be mapped to integers 1-5. However, the reason you can't directly determine distance without mapping to numerical values is because there's no intrinsic distance between the nominal values. Strongly Disagree
could be two units away from Disagree
, and maybe Neutral
is a billion units away from Agree
.
Nominal: If you measure performance using words like "good", "decent", "fine" where there's no clear ordering, then distance makes no sense.