Assume data which is labeled $y_i \in \left\{ 1, 2, 3, \ldots, 9, 10\right\}$.

Assume the labels are ordered, namely, given $y_i = 10$ to estimate $\hat{y}_{i} = 1$ is much worse than $\hat{y}_{i} = 9$.

I am looking for a loss function to take that into consideration and be usable both for deep learning, namely gradient friendly, and decision trees (XGBoost, Scikit-Learn).

I'd be happy even to be able to set the weighted loss by a matrix:

$$L_{i, j} = loss(y_i, \hat{y}_{j})$$

Namely the loss weight (Or multiplier) will be set by a matrix.

Is there a way to achieve this?
What other approaches would you use in that case?


2 Answers 2


A non-deep-learning approach to this kind of problem is ordinal regression, which serves precisely the purpose you described.

To apply the idea of ordinal regression to deep learning, there are at least 2 approaches:

  • $\begingroup$ I see. Am I wrong that all those methods won't let me set the cost of the error per pair? $\endgroup$ Dec 2, 2022 at 18:57

To sketch one approach, you may define the loss as:

$$ loss \left( \boldsymbol{y}, \hat{\boldsymbol{y}} \right) = \left( w + 1 \right) \operatorname{CE} \left( \boldsymbol{y}, \hat{\boldsymbol{y}} \right), \quad w \left( \boldsymbol{y}, \hat{\boldsymbol{y}} \right) = \frac{ \left| \arg \max_{i} \boldsymbol{y} - \arg \max_{i} \hat{\boldsymbol{y}} \right| }{K - 1} $$


  • $K$ - The number of classes (2 for Binary Classification).
  • $\boldsymbol{y}$ - A vector in $\mathbb{R}^{K}$ which is the ground truth probabilities per class.
  • $\hat{\boldsymbol{y}}$ - A vector in $\mathbb{R}^{K}$ which is the estimation of probabilities per class of the classifier.
  • $\operatorname{CE} \left( \cdot, \cdot \right)$ - The Cross Entropy loss function.
  • $\arg \max_{i} \hat{\boldsymbol{y}}$ - Extracts the index of the class with the highest probability. Basically the class represented by the vector.

In the function $ w \left( \boldsymbol{y}, \hat{\boldsymbol{y}} \right) $ you may raise the distance between the class indices as you want.

I think it should work well for you in the context of Deep Learning.

For the matrix case, you may use the same loss as above and set:

$$ w \left( \boldsymbol{y}, \hat{\boldsymbol{y}} \right) = {L}_{\arg \max_{i} \boldsymbol{y}, \arg \max_{i} \hat{\boldsymbol{y}}} $$

Where in my convention $ \boldsymbol{y} $ and $ \hat{\boldsymbol{y}} $ are vectors of the discrete distribution over classes.

One simple extension would be:

$$ w \left( \boldsymbol{y}, \hat{\boldsymbol{y}} \right) = \frac{ {\left| \arg \max_{i} \boldsymbol{y} - \arg \max_{i} \hat{\boldsymbol{y}} \right|}^{p} }{K - 1} $$

Where $ p \geq 1 $.

  • $\begingroup$ Can I use arbitrary loss per pair of class? Like in the matrix? $\endgroup$ Dec 2, 2022 at 7:01
  • $\begingroup$ @GeorgeIrwin, I added something about the matrix case. $\endgroup$
    – Royi
    Dec 2, 2022 at 7:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.