Custom loss function for regression

Question

I am trying to write a custom loss function for a machine learning regression task. What I want to accomplish is following:

• Reward higher preds, higher targets
• Punish higher preds, lower targets
• Ignore lower preds, lower targets
• Ignore lower preds, higher targets

All ideas are welcome, pseudo code or python code works good for me.

This is what I tried so far, it does not work so well I think it is because it does not take high targets into account (just high preds):

def mae_high(inp, targ):
    inp, targ = flatten_check(inp, targ)
    thresh = np.percentile(inp.detach().numpy(), 50)
    mask = inp > thresh
    high_preds = torch.masked_select(inp, mask)
    high_targ = torch.masked_select(targ, mask)
    return torch.abs(high_preds - high_targ).mean()

Answer 1

First I have to stress that this custom loss is very unusual; I would rethink the domain question to double-check the rationale.

Anyway, the pseudo-code should do:

def custom_loss(pred, target):
    threshold = <define the threshold>
    if pred < threshold:
        return 0  # if pred is 'low', we don't care
    # otherwise, pred must be 'high'; write custom logic to scale the loss
    # for example:
    if target > threshold:
        reward = compute_high_loss(pred, target)  # <- note this has to return a negative number as reward
    else:
        reward = compute_low_loss(pred, target)  # <- this has to return a positive number as loss
    return reward

The ugly compute_high_loss() and compute_high_loss() logic is necessary otherwise the model will keep predicting low (which has 0 loss) as @Dave pointed out. That said, I strongly suggest you to double-check your rationale on why you need such a strange loss function.

Answer 2

Not exactly sure what you want to achieve and in what kind of setting. There are some well-known loss functions which you might have a look at.

One option is the Huber-Loss which avoids very large residuals for "high" values and thus can lead to a more balanced prediction. It is a mix of L1 and L2 loss.

Another more flexible loss function is the "fair loss", which can be tuned to some extent as far as I remember (it is not well documented).

If Huber is just the opposit of what you want, you could try to "flip it around" and assign L1 loss logic to "low values" and L2 to high values.

I tried to implement both Huber Loss and Fair Loss in XGBoost.

Answer 3

A bit late to the party, but I had a similar challenge. I wanted my model to favour low prediction errors for high values of the target variable. I can't tell if this is what you are trying to achieve, but I came across this paper: https://link.springer.com/content/pdf/10.1007/s10994-020-05900-9.pdf. Perhaps that's a good starting point.

I also experimented a bit with a custom loss function that is a simple squared error term, but scaled with the value of the target: loss=y*(y_pred-y)^2. I reasoned that the model would try to fit high target values more accurately in this case. Another option might be to take the square root of the scaling factor. I can't say much about the properties of these custom loss function, however, so you have to look into that yourself.