0
$\begingroup$

I have a classification problem and I am producing a confusion matrix. Ideally one wants to get all results in the diagonal. I get quite many points around diagonal for different algorithms. Still for my use-case I want to favor algorithms that underpredict the class (I have ordinal data) and not overpredict.

Is there a metric that can measure under and overprediction and rate those errors with a different weight? The typical accuracy, precision terms assume that all mistakes are the same.

Of course I can try to implement my own metric but I am quite sure that I am not the first that is having this issue.

Any metric available that you know already? Thanks Alex

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You are incorrect when you say accuracy and precision assume the same mistakes. They are quite different in nature as to when they are applied for separate use-cases.

For more details on how they are different - https://towardsdatascience.com/accuracy-precision-recall-or-f1-331fb37c5cb9

As per your situation, you could opt for precision as you are looking only for the right answers, in case of a spam email detection.

There is no metric as such which calculates how much your model underfits or overfits but a genera idea can be made when you compare the validation curve and the training curve.

Improve this answer
$\endgroup$
3
  • $\begingroup$ My bad phrasing. Accuracy and prediction do not differentiate on the type of error. Also I am not sure I want to see if my model overfits or underfits but rather if the model has some tendency (i.e more frequently to predict a class higher of what it is) (i.e more rarely predicts a class lower of what it is) $\endgroup$
    – Alex P
    Jan 6, 2021 at 8:43
  • $\begingroup$ Could you elaborate on "class higher" and "class lower"? $\endgroup$
    – Academic
    Jan 7, 2021 at 1:20
  • $\begingroup$ sure! Lets say that you have the following classes that refer to transmission speeds. Class 1,2,3,4,5,6. Class 1 is the lowest speed and Class 6 is the highest. Assume that we wanted to predict class 3 but there are two algorithms. Algorithm A: Predicts 2 (1 less) and Algorithm B: predicts 4 ( 1 more). Accuracy and precision will count those errors equally (u have just missed the right class). For a real use-case algorithm A is more useful (shall I elaborate the why?) . Algorithm B at the other side (predicting more) can have quite devastating results. $\endgroup$
    – Alex P
    Jan 11, 2021 at 8:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.