# Is Pearson correlation a good loss function?

I want to do a data science project. I want to use price history to predict future prices.

I want to use correlation(y, y_pred) as my loss function but I found it's hard to calculate first deter, and second deter.

Has anyone used correlation as loss function, and is it good?

I think Dave's answer points out the most pressing issues:

• translational invariance
• Absolute scale invariance

In Tensorflow we can define our correlation function:

class CorrLoss(tf.keras.losses.Loss):

def call(self, y_true, y_pred):
res_true = y_true - tf.reduce_mean(y_true)
res_pred = y_pred - tf.reduce_mean(y_pred)
cov = tf.reduce_mean(res_true * res_pred)
var_true = tf.reduce_mean(res_true**2)
var_pred = tf.reduce_mean(res_pred**2)
sigma_true = tf.sqrt(var_true)
sigma_pred = tf.sqrt(var_pred)
return - cov / (sigma_true * sigma_pred)


And quickly whip up a simple linear model:

model = tf.keras.Sequential([
layers.Dense(input_shape=[1,], units=1)
])


And a data set that is learnable by this model:

x = tf.random.normal((1000,))
y = 5 * x + 10 + tf.random.normal((1000,))


Training with our choice of loss function, model, and data, we can visually understand that correlation alone is not sufficient. As Dave describes, least squares is often effective.

Mostly for my own amusement, I considered if maximizing $$\mathbb{E}[Y \hat Y]$$ would fare any better than maximizing Pearson's correlation.

Here is the custom loss function:

class ProdLoss(tf.keras.losses.Loss):

def call(self, y_true, y_pred):
return -tf.reduce_mean(y_true * y_pred)


The following is a close success over the consistently-horrible choice of correlation:

And often it would look better, but it wasn't reliable! It would also often look like this:

Interestingly, the product moment will tend to ignore the true values by making the predicted values extreme. I noticed this by taking the same problem and increasing the number of epochs to $$10^4$$.

Thus the correlation and mixed moment are unreliable loss functions for achieving $$Y \approx \hat Y$$.

• Note that I used epochs=1000, which is plenty for this problem when using mean squared error as loss. Sep 18, 2022 at 0:37
• +1 The shame of it is that both will have the OLS solution as a possible solution…among the infinitely many other solutions that minimize the loss.
– Dave
Sep 18, 2022 at 0:46
• I use some related techniques here: stats.stackexchange.com/a/589221/69508 Sep 19, 2022 at 1:10
• I asked a related type of question here: stats.stackexchange.com/questions/589160/… Sep 19, 2022 at 1:13

UPDATE (I WAS WRONG)

Maximizing correlation misses a lot and makes for a terrible function to optimize.

For instance, correlation will not detect if you consistently predict too high or too low.

For every real $$a$$: $$corr(y, \hat y) = corr(y, a+\hat y)$$

In fact, for any real $$a$$ and positive $$b$$:

$$corr(y,\hat y) = corr(y,a+b\hat y)$$

Concretely, if your true values are $$y=(1,2,4,6,5)$$, $$\hat y=(11, 21, 41, 61,51)$$ makes for terrible predictions but does have a perfect correlation with $$y$$. (In this example, $$a=1$$ and $$b=10$$.)

Minimizing square loss implies a maximization (not minimization) of correlation between observed and predicted values, but the reverse implication does not apply.

However, since minimizing square loss implies a maximization of correlation (at least in linear models), you can minimize square loss and get maximized correlation. Correlation will not be perfect unless square loss is zero, but you can still maximize the correlation in the sense that no other regression coefficients will do better (though some will result in equal correlation but worse or equal square loss).

Minimizing square loss is what happens in ordinary least squares linear regression and is completely standard in statistics and machine learning.

• My simulations on the statistics Stack give additional examples of the dangers of using correlation or (worse) squared correlation as a measure of model performance.
– Dave
Sep 18, 2022 at 0:57

Correlation does not make a useful loss function for many reasons. One reason is that correlation only measures how linearly related two variables are. A model can have a strong linear relationship between observed and predicted values and still make terrible predictions.

For example:

observed = [1, 2, 3, 4]
predicted = [101, 102, 103, 104]

Those observed and predicted values are perfectly correlated (i.e., r=1) but the predictions are way off.