1
$\begingroup$

I'm looking for an ML evaluation metric that would work well with imbalanced and ordinal multiclass datasets:

Imagine you want to predict the severity of a disease that has 4 grades of severity where 1 is mild and 4 represent the worse outcome. Now, this dataset would realistically have the vast majority of patients in the mild zone (classes 1 or 2) and fewer in classes 3 and 4. (Imbalanced/skewed dataset).

Now in the example, a classifier that predicts a grade 4 as grade 1, should be penalised more than a classifier that predicts a grade 4 as grade 3 etc. (Ordinal class).

If I use MCC, Cohen's K etc. I will be able to account for the imbalance in the dataset but not for the ordinal nature of its class. Would you know if there is a metric that would account for both or if there is a way to modify/combine metrics so that both aspects of the dataset would be taken into account? (If possible using Python but also other languages or a mathematical explanation would work)

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

I would consider using a regression evaluation measure such as RMSE:

  • It takes into account the ordinal nature of the values since it's based on the error between the predicted and true value. This means that an error between 2 and 4 is penalized more than an error between 2 and 3 for instance.
  • It would also take into account the imbalance but only to the extent that errors between classes 1,2 and 3,4 are more likely to be large, so it might not be perfect.

If more refinement is needed, a solution would be to define a custom measure of errors, i.e. define the penalty for every possible error. For example:

  • errors 1->2, 2->1, 2->3, 3->4 counted as 1 (small error)
  • errors 1->3, 2->4, 3->2, 4->3 counted as 5 (moderate error)
  • errors 1->4, 4->2, 3->1 counted as 10 (serious error)
  • errors 4->1 counted as 20 (critical error)

Above my notation "x->y" means true value x wrongly predicted as y.

Of course I'm making up these values, it's just to show that one can go as fine-grained as needed depending on the task.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Theank you Erwan, I was actually thinking of using both MCC and RMSE so I will be able to account for imbalanced dataset and also the ordinal class. I'd like to be able to combine those 2 in a single custom metric. Would you have any idea of how to do that? $\endgroup$
    – Fabio Magarelli
    Feb 4, 2022 at 9:11
  • $\begingroup$ @FabioMagarelli I don't think there's any standard method for this. The first thing that comes to mind is to simply multiply the two values: since MCC should be positive, it gives a basic combination of the two measures. $\endgroup$
    – Erwan
    Feb 4, 2022 at 14:13
1
$\begingroup$

There are variations of Cohens Kappa that are meant to be applied to ordinal scales. Sklearn cohen_kappa has options for linear and quadratic.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi Michael, would you know if the weight is applied to the distance between prediction and real value? Thus penalising more a prediction that has higher distance from the real value than one with a smaller distance? $\endgroup$
    – Fabio Magarelli
    Feb 7, 2022 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.