Currently my classification model is doing too well on all of the train, validation, and test datasets. I'm assuming there is a data leakage in the features, and therefore I've computed the correlation coefficient of each feature with a target feature.
A few columns have 0.93, 0.927, 0.90 correlation with the target, meaning they have high predictive power.
How high of a correlation with a target is considered data leakage and therefore the column is be removed?
You should not remove features just because their correlation to the target is high. Such high correlation is a sign of potential target leakage. You should understand why their correlation is high.
If the highly-correlated feature is available at inference time and there is a causal relationship between it and the target, then the feature has high predictive power and you should keep it.
On the other hand, if the feature is not available at inference time or there is no causal relation at all, then this sounds like target leakage and you should remove it. This article has some nice examples of each case.
Another thing you may also pay attention to is correlation among features, which may be a bad thing depending on your model (i.e. linear model) and whether you want to interpret it (i.e. importance of each feature when predicting). This DS stackexchange question addresses such problem.
