in my dataset, I am having plenty of features and two features are highly correlated to each other and giving same impact on target variable, in this case which feature we need to select in order to build model and what strategy we have to use to select any one of the features

  • $\begingroup$ Why not pick both? $\endgroup$
    – Dave
    Jul 13, 2021 at 20:14
  • $\begingroup$ Its not a good way to take both features which are highly correlated to each other ,because each features has to be independent to each other not dependent . $\endgroup$ Jul 14, 2021 at 8:52
  • $\begingroup$ According to what? The Gauss-Markov theorem makes no such assumption, for instance. $\endgroup$
    – Dave
    Jul 14, 2021 at 9:31

1 Answer 1


First let me answer your specific question: If you want to decide which feature of two highly correlated, high impact features I would look at the following additional attributes of your features:

  • How is the data quality or amount of data? Is one better or higher than the other? Choose this one.
  • Is it in any way harmful to remove one of the features? If yes, keep the one with lees harm (for example accuracy drops, other calculations become complicated or maybe you need one of the features for your output)

However I want to ask you to consider the following before removing any of those features:

  1. Is having both features in any way harmful to your accuracy? For example if your features correlate with each other but each also correlates with your target it might still be fine to use both. Also depending on your algorithm it is not harmful at all to use both. Some algorithms are more sensitive to multicolinearity than others.
  2. Do you safe considerable time when training if you remove one of your features? If so than go ahead and remove the one by looking at the example attributes I listed above
  3. What is important for you? Coefficients/p-values or prediction accuracy? Multicolinearity can be harmful for coefficients and the interpretability of your model but still yield good prediction accuracy. Multicolinearity between features doesn't necessarily mean that you have bad predictions.
  4. Do you have a problem of high dimensionality? If so removing high-correlation features reduces your number of dimensions and is therefor beneficial for your algorithm

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.