4
$\begingroup$

I am attempting to build a sequential model with Keras (Tensorflow backend) that has multiple outputs. My targets are proportions of a whole so each observation is an array like [0.5, 0.25, 0.15, 0.1]. The sum of each observation's array equals 1.

When I train the model, minimizing mean squared error, I get OK results. But looking over the validation results, it's clear that mean squared error may not be the best loss function for my problem. It is important for me that the sequences within y_true and y_pred are highly correlated.

As an example, let's say I have one observation [10%, 20%, 30%, 40%]. And each of the following are potential predictions for this observation:

  1. [20%, 30%, 20%, 30%]
  2. [20%, 10%, 40%, 30%]
  3. [00%, 30%, 20%, 50%]
  4. [00%, 10%, 40%, 50%]
import numpy as np
from sklearn.metrics import mean_squared_error

y_true   = np.array([[0.1, 0.2, 0.3, 0.4]])
y_pred_0 = np.array([[0.2, 0.3, 0.2, 0.3]])
y_pred_1 = np.array([[0.2, 0.1, 0.4 ,0.3]])
y_pred_2 = np.array([[0.0, 0.3, 0.2, 0.5]])
y_pred_3 = np.array([[0.0, 0.1, 0.4, 0.5]])

preds = [y_pred_0, y_pred_1, y_pred_2, y_pred_3]

for i, pred in enumerate(preds):
    corr = np.corrcoef(y_true, pred)[0,1]
    mse = mean_squared_error(y_true, pred)
    print(f'y_pred_{i}: corr={corr:.2f}, mse={mse:.2f}')

Gives us:

y_pred_0: corr=0.45, mse=0.01
y_pred_1: corr=0.60, mse=0.01
y_pred_2: corr=0.87, mse=0.01
y_pred_3: corr=0.98, mse=0.01

So while each of these four predictions has the same error, the fourth is most preferable to me because the sequence within the predicted array is most correlated with the observation's sequence.

I've found others who have used a modified correlation coefficient function as a loss function. But if I optimize on correlation coefficient, an optimal prediction for the example observation might be [0%, 1%, 2%, 3%] since the two sequences are perfectly correlated. This doesn't work since the mean error is so large.

So I need to optimize both for high correlation and a small error. I'm unable to find anywhere where this kind of problem has been solved. Is there a way I can optimize for both of these objectives? Particularly in the Keras framework?

$\endgroup$
1
$\begingroup$

MSE uses the euclidean distance function. You could simply use the Pearson correlation coefficient as similarity measure, so define a new distance function that decreases when the coefficient increases ($exp(-\rho)$ for instance).

Another possibility is to use the Wasserstein distance (or Wasserstein metric, or earth mover's distance). See the definitions on Wikipedia, especially this one which is not too complicated.
This metric is designed to compute a distance between distributions (continuous) or histograms (binned / discrete). It does not relate to correlation, but rather to the shape of the vector when plotted as an histogram. This is not what you asked for but it might be what you're actually looking for.

$\endgroup$
  • $\begingroup$ Hi Romain. I had considered a modified correlation coefficient but there's the possibility the model would settle on a prediction like [0%,1%, 2%, 3%] as explained in the question so I don't think correlation alone will solve this problem. I also tried scipy's implementation of Wasserstein distance on your suggestion. The distance metric for each of my 4 predictions above are: y_pred_0=0.05, y_pred_1=0.0, y_pred_2=0.05, y_pred_3=0.1. Does this seem reasonable? Intuition would say the metric should decrease with each prediction. $\endgroup$ – Jeremy Doyle Dec 22 '19 at 14:03
1
$\begingroup$

Customize your own loss function

For example:

import keras.backend as K def customLoss(y_true,y_pred): corr = np.corrcoef(y_true, pred)[0,1] mse = mean_squared_error(y_true, pred) return (mse+corr)

And than simply

model.compile(loss=customLoss, optimizer = .....)

You could add some weights, penalties etc...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.