Removing features that correlate with the target label

Question

I know that it is better to remove correlated features, but what if the feature correlates with the target label? So, it gives the most information for prediction? Do I need to remove features that correlate with the target label?

Answer 1

Features related to the target variable are features that help you predict the target.

No, don’t remove those!

Feature selection is surprisingly fraught with problems, but removing a variable from a predictive model because it has an obvious relationship with the outcome seems like an especially poor approach.

Answer 2

The idea behind removing features that correlate with each other is that, by increasing the independence of each feature in an observation a better model can be created. There is another thread that goes into further detail on the reasons why here.

However, the odd part of your question is that you are comparing the correlation between the dependent and independent variables. This type of comparison is usually done after a model is constructed in order to prevent to assess the predictive strength of the model. To compare the target label, the label you wish to predict, with the other variables before this is premature and will likely result in weakening your model.

When testing for correlated features, it is best that you first remove the target label, before check for the correlations and determining which to remove if any. Then if you want to remove features further you can perform principal component analysis on the data. From there you can run the data through say a logistic regression model and select the most important features from there, like mentioned here.