Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
E.g. In detriment of a smaller mean error, I want to have fewer big mistakes
I'm working on a time series forecasting task and in some specific cases I don't need perfect accuracy, but the network cannot by any mean miss by a lot.
Any suggestions of loss functions or other methods to solve this issue?
As you increase the harshness of big misses, you make the model less willing to miss big.
For instance, absolute loss considers missing by $2$ to be twice as bad as missing by $1$, but square loss considers missing by $2$ to be four times as bad as missing by $1$!
Imagine if you use cubic loss:
$$L_3(y,\hat{y}) = \sqrt[3]{\sum\big\vert y_i - \hat{y}_i \big\vert^3}$$
Imagine if you use quintic loss:
$$L_5(y,\hat{y}) = \sqrt[5]{\sum\big\vert y_i - \hat{y}_i \big\vert^5}$$
This gets at the general form of $L_p$ loss, which I'm guessing you can guess before reading what I type.
$$L_p(y,\hat{y}) = \sqrt[p]{\sum\big\vert y_i - \hat{y}_i \big\vert^p}$$
This relates to the idea of $L^p$ norms.
At the extreme of $L^p$ norms, you can set $p=\infty$, which is equivalent to just looking at the maximum. If you use $L_{\infty}$ loss, you minimize the greatest error.
I've wanted to play around with $L_{\infty}$ loss for a while and would be very curious what happens if you choose this loss function. I do, however, think this would be a very extreme loss function.
(Remember that $RMSE$, $MSE$, and $SSE$ all have the same $argmin$; by similar logic we are free to take the $p^{th}$ roots to get the cubic and qunitic loss functions I gave.)
You could consider something like the relative error $$L(y,\hat{y}) = \sqrt[]{\sum\big\vert (y_i - \hat{y}_i) / y_i \big\vert^2}$$
