2
$\begingroup$

I'm dealing with a regression prediction challenge where the evaluation metric is (pearson) correlation. However, I have the impression that this metric is kind of arbitrary. While I can keep the RMSE stable the correlation can have great variety.

Could someone please explain this metric and how to optimise for it?

Improve this question
$\endgroup$
2
  • $\begingroup$ I'm slightly confused; if you don't think this is an appropriate metric, why are you using it? $\endgroup$
    – Emre
    Jan 24, 2016 at 6:37
  • $\begingroup$ @Emre because the challenge organizers defined it to be their evaluation metric $\endgroup$
    – spore234
    Jan 24, 2016 at 9:40

1 Answer 1

Reset to default
1
$\begingroup$

The organizers might deem the direction of predicted change more important than the magnitude, i.e., it is more important that your prediction is high when the known value is high (and vice versa) than to get as close to the known value as possible. The measurements might be noisy anyway.

One, fairly robust way of optimizing for it would be by grid-searching for the local optimum, like in this QA.

However, you should also take note that algorithms tweak ther internal parameters when fitting according to some loss function. Some algorithms accept custom cost functions and derivatives, yet some implementations don't. Information-theoretic measureas are standard in classification while MSE in regression.

Theoretically, you should be able to tell your Random Forest (or another algorithm accordingly) what the optimal split is as a function of pearson correlation.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I did a CV gridsearch for the hyperparameters of a random forest and my results are very unstable. Correlation ranges between 0.15 and 0.4, while RMSE is between 0.45 and 0.41 for the hyperparameters. When I predict on a hold-out set the correlation is far off the CV values. Hyperparameters that perform well in gridsearch perform badly on the holdout set and vice versa. $\endgroup$
    – spore234
    Jan 25, 2016 at 7:44
  • $\begingroup$ Indeed, please see the updated answer. $\endgroup$
    – K3---rnc
    Jan 25, 2016 at 17:40
  • $\begingroup$ The problem is that regression minimizes a cost function (presumably using least squares in your case) and the cost function is not based on the correlation, but also includes adjustments to the y intercept. Your grid search is effectively interfering with your cost function minimization. If you are coding this yourself, I suggest choosing the Pearson correlation as the cost function within the algorithm. Thus the y intercept (w0 or bias weight) won't be fit and will reduce the bias of the problem. $\endgroup$
    – AN6U5
    Jan 26, 2016 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.