# In supervised learning, why is it bad to have correlated features?

I read somewhere that if we have features that are too correlated, we have to remove one, as this may worsen the model. It is clear that correlated features means that they bring the same information, so it is logical to remove one of them. But I can not understand why this can worsen the model.

• This rule applies more strongly in some models and analysis than others. Any chance you could add some context to "I read somewhere" - e.g. was it relation to training a specific model? – Neil Slater Nov 7 '17 at 14:46
• Correlated features will not necessarily worsen a model. Removing correlated features helps to infer meaning about the features. – Hobbes Nov 7 '17 at 14:48

Correlated features in general don't improve models (although it depends on the specifics of the problem like the number of variables and the degree of correlation), but they affect specific models in different ways and to varying extents:

1. For linear models (e.g., linear regression or logistic regression), multicolinearity can yield solutions that are wildly varying and possibly numerically unstable.

2. Random forests can be good at detecting interactions between different features, but highly correlated features can mask these interactions.

More generally, this can be viewed as a special case of Occam's razor. A simpler model is preferable, and, in some sense, a model with fewer features is simpler. The concept of minimum description length makes this more precise.

• Numerical stability aside, prediction given by OLS model should not be affected by multicolinearity, as overall effect of predictor variables is not hurt by presence of multicolinearity. It is interpretation of effect of individual predictor variables that are not reliable when multicolinearity is present. – Akavall Nov 7 '17 at 22:23

(Assuming you are talking about supervised learning)

Correlated features will not always worsen your model, but they will not always improve it either.

There are three main reasons why you would remove correlated features:

• Make the learning algorithm faster

Due to the curse of dimensionality, less features usually mean high improvement in terms of speed.

If speed is not an issue, perhaps don't remove these features right away (see next point)

• Decrease harmful bias

The keyword being harmful. If you have correlated features but they are also correlated to the target, you want to keep them. You can view features as hints to make a good guess, if you have two hints that are essentially the same, but they are good hints, it may be wise to keep them.

Some algorithms like Naive Bayes actually directly benefit from "positive" correlated features. And others like random forest may indirectly benefit from them.

Imagine having 3 features A, B, and C. A and B are highly correlated to the target and to each other, and C isn't at all. If you sample out of the 3 features, you have 2/3 chance to get a "good" feature, whereas if you remove B for instance, this chance drops to 1/2

Of course, if the features that are correlated are not super informative in the first place, the algorithm may not suffer much.

So moral of the story, removing these features might be necessary due to speed, but remember that you might make your algorithm worse in the process. Also, some algorithms like decision trees have feature selection embedded in them.

A good way to deal with this is to use a wrapper method for feature selection. It will remove redundant features only if they do not contribute directly to the performance. If they are useful like in naive bayes, they will be kept. (Though remember that wrapper methods are expensive and may lead to overfitting)

If your model needs to be interpretable, you might be forced to make it simpler. Make sure to also remember Occam's razor. If your model is not "that much" worse with less features, then you should probably use less features.

Sometimes correlated features -- and the duplication of information that provides -- does not hurt a predictive system. Consider an ensemble of decision trees, each of which considers a sample of rows and a sample of columns. If two columns are highly correlated, there's a chance that one of them won't be selected in a particular tree's column sample, and that tree will depend on the remaining column. Correlated features mean you can reduce overfitting (through column sampling) without giving up too much predictive quality.

Making a decision should be done on the minimum necessary variables to do so. This is, as mentioned above, the formalization of Occam's razor with minimum description length above. I like that one.

I would tend to characterize this phenomena in something like a HDDT to mean the most efficient tree that makes no spurious decision based on available data, and avoiding all instances of decisions that may otherwise have been made on multiple data points without understanding that they were correlated.

• Regarding datascience.stackexchange.com/users/38887/valentin-calomme comment: "Correlated features will not always worsen your model, but they will not always improve it either." I don't see or can't think of where having high correlation between variables doesn't make your model worse. At least in the sense that, given the choice: I'd rather train a network with less correlated features. Anything other than that is functionally and provably worse. Are there instances when this isn't true? – tjborromeo Aug 7 '18 at 10:07

In perspective of storing data in databases, storing correlated features is somehow similar to storing redundant information which it may cause wasting of storage and also it may cause inconsistent data after updating or editing tuples.

If we add so much correlated features to the model we may cause the model to consider unnecessary features and we may have curse of high dimensionality problem, I guess this is the reason for worsening the constructed model.

In the context of machine learning we usually use PCA to reduce the dimension of input patterns. This approach considers removing correlated features by someway (using SVD) and is an unsupervised approach. This is done to achieve the following purposes:

Although this may not seem okay but I have seen people that use removing correlated features in order to avoid overfitting but I don't think it is a good practice. For more information I highly recommend you to see here.

Another reason is that in deep learning models, like MLPs if you add correlated features, you just add unnecessary information which adds more calculations and parameters to the model.

The answer to this question depends greatly upon the purpose of the model. In inference, highly correlated features are a well-known problem. For example, two features highly correlated with each other and with y, might both come out as insignificant in an inference model, potentially missing an important explanatory signal. Therefore, in inference it is generally recommended to thin them out.

If your supervised learning is for prediction, the answer - counter to conventional wisdom - is usually the opposite. The only reason to remove highly correlated features is storage and speed concerns. Other than that, what matters about features is whether they contribute to prediction, and whether their data quality is sufficient.

Noise-dominated features will tend to be less correlated with other features, than features correlated with y. Hence, as mentioned above in the example by Valentin, thinning out the latter will increase the proportion of the former.

In particular, methods like random forests and KNN treat all features equally, so thinning out correlated features directly reduces their signal-to-noise ratio.

Methods that auto-select features like single trees, "pure" lasso, or neural networks, might be less affected. But even then, other than longer computing time, there is rarely anything to lose prediction-wise from keeping correlated features in the mix.