# Creating a custom loss function for an image classification model where the label matters

I have the following dataset of images, where we can see the image distribution of labels below.

I want to construct a loss function that, on the one hand, outputs probabilities for a specific class and, on the other hand, rewards the model for being closer to the label.

For example, if my label is 20, and I predicted 19, then 19 is better than 18 or 22, but it is the same as 21.

At first, I trained a model where I made specific labels into a set of a smaller number of classes. For example, the confusion matrix shows which labels I put in which class.

Implementing it as an regression problem where the labels distribution is like that is not really atteinable.

Ultimately, I want to process, say, eight images in a row and add up all of the various projections. I care about the total prediction, thus I must figure out how to account for the higher labels.

1. Calculate softmax probabilities for each image's output predictions (using 30 classes) 2.⁠ Calculate the expected label value as a weighted sum of the possible label values, where the weights are the softmax probabilities.
2. Use the difference between the expected label value and the actual label as part of loss function. (using L1) with applying mean before returning the loss function

Another angle I thought about is to use a Weighted Cross-Entropy for the imbalanced with a distant based componenet

Any suggestions on how to insert into the cross-entropy loss function a regression angle?

[. I thought about using the expectation of the data and calculating

The image distribution

A way to take the ordering of the classes into account is to use the ranked probability score (RPS), a different choice of loss function for multi-class classification. It evaluates the predicted probabilities in the form of the cumulated distribution function they define over the classes. More precisely, for $$n$$ classes, predicted probabilities $$p_1, \ldots, p_n$$ and an observation $$y \in \{ 1, \ldots, n\}$$ we have $$\text{RPS}(p_1, \ldots, p_n, y) := \sum_{i=1}^{n} \left( \sum_{j=1}^{i} p_j -1_{i \succeq y} \right)^2.$$ So a prediction will be better if it puts more probability near the observation. For more details you can check out the answer to Performance Metrics for Classification Models with an Ordinal Response Variable on CrossValidated SE.