Loss function for multivariate regression where relationship between outputs matters

Question

I am attempting to build a sequential model with Keras (Tensorflow backend) that has multiple outputs. My targets are proportions of a whole so each observation is an array like [0.5, 0.25, 0.15, 0.1]. The sum of each observation's array equals 1.

When I train the model, minimizing mean squared error, I get OK results. But looking over the validation results, it's clear that mean squared error may not be the best loss function for my problem. It is important for me that the sequences within y_true and y_pred are highly correlated.

As an example, let's say I have one observation [10%, 20%, 30%, 40%]. And each of the following are potential predictions for this observation:

1. [20%, 30%, 20%, 30%]
2. [20%, 10%, 40%, 30%]
3. [00%, 30%, 20%, 50%]
4. [00%, 10%, 40%, 50%]

import numpy as np
from sklearn.metrics import mean_squared_error

y_true   = np.array([[0.1, 0.2, 0.3, 0.4]])
y_pred_0 = np.array([[0.2, 0.3, 0.2, 0.3]])
y_pred_1 = np.array([[0.2, 0.1, 0.4 ,0.3]])
y_pred_2 = np.array([[0.0, 0.3, 0.2, 0.5]])
y_pred_3 = np.array([[0.0, 0.1, 0.4, 0.5]])

preds = [y_pred_0, y_pred_1, y_pred_2, y_pred_3]

for i, pred in enumerate(preds):
    corr = np.corrcoef(y_true, pred)[0,1]
    mse = mean_squared_error(y_true, pred)
    print(f'y_pred_{i}: corr={corr:.2f}, mse={mse:.2f}')

Gives us:

y_pred_0: corr=0.45, mse=0.01
y_pred_1: corr=0.60, mse=0.01
y_pred_2: corr=0.87, mse=0.01
y_pred_3: corr=0.98, mse=0.01

So while each of these four predictions has the same error, the fourth is most preferable to me because the sequence within the predicted array is most correlated with the observation's sequence.

I've found others who have used a modified correlation coefficient function as a loss function. But if I optimize on correlation coefficient, an optimal prediction for the example observation might be [0%, 1%, 2%, 3%] since the two sequences are perfectly correlated. This doesn't work since the mean error is so large.

So I need to optimize both for high correlation and a small error. I'm unable to find anywhere where this kind of problem has been solved. Is there a way I can optimize for both of these objectives? Particularly in the Keras framework?

Answer 1

MSE uses the euclidean distance function. You could simply use the Pearson correlation coefficient as similarity measure, so define a new distance function that decreases when the coefficient increases ($exp(-\rho)$ for instance).

Another possibility is to use the Wasserstein distance (or Wasserstein metric, or earth mover's distance). See the definitions on Wikipedia, especially this one which is not too complicated.
This metric is designed to compute a distance between distributions (continuous) or histograms (binned / discrete). It does not relate to correlation, but rather to the shape of the vector when plotted as an histogram. This is not what you asked for but it might be what you're actually looking for.

Answer 2

Customize your own loss function

For example:

import keras.backend as K def customLoss(y_true,y_pred): corr = np.corrcoef(y_true, pred)[0,1] mse = mean_squared_error(y_true, pred) return (mse+corr)

And than simply

model.compile(loss=customLoss, optimizer = .....)

You could add some weights, penalties etc...