0
$\begingroup$

I'm working with a multi-class classification problem, using Keras Sequential models. In my dataset, the output class has one of the following values: (1, 2, 3, 4, 5).

Yes, I have already tried one-hot-encoding the output and using categorical_crossentropy as a loss function and softmax in the output layer.

However, I don't think that softmax and categorical cross entropy is the right choice in my case. In my dataset, the output classes have a certain "discreteness" (or scale). Class 1 is the "worst" and class 5 is the "best".

Let's say if on a specific input the model predicts 2 and the true class is 1, it's a much better prediction as when the model predicts class 5 and true class is 1.

I would like, that the loss function would take these "minor errors" into consideration and not treat all errors the same way.

To sum up, I'm wondering what my options are in terms of loss function and activation of the output layer for my given problem.

UPDATE: I have now tried to use mse as a loss function and replace my output layer with this:

model.add(Dense(1, activation='linear'))

The new method produces much better results, but I'm still wondering if there are any other options for my problem. Maybe a custom loss function?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ You can do ordinal regression as you happen to have ordinal dependent variable. $\endgroup$
    – naive
    Commented Mar 9, 2020 at 12:38
  • $\begingroup$ This is an ordinal regression, not a classification. When you treat it as an ordinal regression, the process should make more sense. $\endgroup$
    – Dave
    Commented Aug 21, 2022 at 20:41

1 Answer 1

Reset to default
0
$\begingroup$

You could try a Quadratic Kappa Metric loss function. This is a summary of what it actually does.

A weighted Kappa is a metric which is used to calculate the amount of similarity between predictions and actuals. A perfect score of 1.0 is granted when both the predictions and actuals are the same. Whereas, the least possible score is -1 which is given when the predictions are furthest away from actuals. In our case, consider all actuals were 0's and all predictions were 4's. This would lead to a QWKP score of -1. The aim is to get as close to 1 as possible. Generally a score of 0.6+ is considered to be a really good score.

The detailed explanation can be found here: https://www.kaggle.com/aroraaman/quadratic-kappa-metric-explained-in-5-simple-steps

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.