6
$\begingroup$

Currently my classification model is doing too well on all of the train, validation, and test datasets. I'm assuming there is a data leakage in the features, and therefore I've computed the correlation coefficient of each feature with a target feature.

A few columns have 0.93, 0.927, 0.90 correlation with the target, meaning they have high predictive power.

How high of a correlation with a target is considered data leakage and therefore the column is be removed?

$\endgroup$

1 Answer 1

11
$\begingroup$

You should not remove features just because their correlation to the target is high. Such high correlation is a sign of potential target leakage. You should understand why their correlation is high.

If the highly-correlated feature is available at inference time and there is a causal relationship between it and the target, then the feature has high predictive power and you should keep it.

On the other hand, if the feature is not available at inference time or there is no causal relation at all, then this sounds like target leakage and you should remove it. This article has some nice examples of each case.

Another thing you may also pay attention to is correlation among features, which may be a bad thing depending on your model (i.e. linear model) and whether you want to interpret it (i.e. importance of each feature when predicting). This DS stackexchange question addresses such problem.

$\endgroup$
2
  • 2
    $\begingroup$ Excellent answer! $\endgroup$ Nov 13, 2023 at 9:14
  • 1
    $\begingroup$ Great answer with 1 key take-home message: it is always the why that matters. $\endgroup$
    – lpounng
    Nov 16, 2023 at 4:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.