I am attempting to build a sequential model with Keras (Tensorflow backend) that has multiple outputs. My targets are proportions of a whole so each observation is an array like [0.5, 0.25, 0.15, 0.1]. The sum of each observation's array equals 1.
When I train the model, minimizing mean squared error, I get OK results. But looking over the validation results, it's clear that mean squared error may not be the best loss function for my problem. It is important for me that the sequences within y_true and y_pred are highly correlated.
As an example, let's say I have one observation [10%, 20%, 30%, 40%]. And each of the following are potential predictions for this observation:
- [20%, 30%, 20%, 30%]
- [20%, 10%, 40%, 30%]
- [00%, 30%, 20%, 50%]
- [00%, 10%, 40%, 50%]
import numpy as np
from sklearn.metrics import mean_squared_error
y_true = np.array([[0.1, 0.2, 0.3, 0.4]])
y_pred_0 = np.array([[0.2, 0.3, 0.2, 0.3]])
y_pred_1 = np.array([[0.2, 0.1, 0.4 ,0.3]])
y_pred_2 = np.array([[0.0, 0.3, 0.2, 0.5]])
y_pred_3 = np.array([[0.0, 0.1, 0.4, 0.5]])
preds = [y_pred_0, y_pred_1, y_pred_2, y_pred_3]
for i, pred in enumerate(preds):
corr = np.corrcoef(y_true, pred)[0,1]
mse = mean_squared_error(y_true, pred)
print(f'y_pred_{i}: corr={corr:.2f}, mse={mse:.2f}')
Gives us:
y_pred_0: corr=0.45, mse=0.01
y_pred_1: corr=0.60, mse=0.01
y_pred_2: corr=0.87, mse=0.01
y_pred_3: corr=0.98, mse=0.01
So while each of these four predictions has the same error, the fourth is most preferable to me because the sequence within the predicted array is most correlated with the observation's sequence.
I've found others who have used a modified correlation coefficient function as a loss function. But if I optimize on correlation coefficient, an optimal prediction for the example observation might be [0%, 1%, 2%, 3%] since the two sequences are perfectly correlated. This doesn't work since the mean error is so large.
So I need to optimize both for high correlation and a small error. I'm unable to find anywhere where this kind of problem has been solved. Is there a way I can optimize for both of these objectives? Particularly in the Keras framework?