Skip to main content
78 votes
Accepted

In supervised learning, why is it bad to have correlated features?

Correlated features in general don't improve models (although it depends on the specifics of the problem like the number of variables and the degree of correlation), but they affect specific models in ...
Ami Tavory's user avatar
  • 1,277
63 votes
Accepted

Does XGBoost handle multicollinearity by itself?

Decision trees are by nature immune to multi-collinearity. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models ...
Sandeep S. Sandhu's user avatar
59 votes
Accepted

Pearson vs Spearman vs Kendall

Correlation is a bivariate analysis that measures the strength of association between two variables and the direction of the relationship. In terms of the strength of the relationship, the value of ...
Pluviophile's user avatar
  • 3,918
40 votes

In supervised learning, why is it bad to have correlated features?

(Assuming you are talking about supervised learning) Correlated features will not always worsen your model, but they will not always improve it either. There are three main reasons why you would ...
Valentin Calomme's user avatar
29 votes

Does XGBoost handle multicollinearity by itself?

I was curious about this and made a few tests. I’ve trained a model on the diamonds dataset, and observed that the variable “x” is the most important to predict whether the price of a diamond is ...
dalloliogm's user avatar
27 votes

Does XGBoost handle multicollinearity by itself?

There is an answer from Tianqi Chen (2018). This difference has an impact on a corner case in feature importance analysis: the correlated features. Imagine two features perfectly correlated, ...
Jiaxiang's user avatar
  • 400
16 votes

In supervised learning, why is it bad to have correlated features?

Why is Multicollinearity a Potential Problem? A key goal of regression analysis is to isolate the relationship between each independent variable and the dependent variable. The interpretation of a ...
Pratik Nabriya's user avatar
16 votes

If A and B are correlated and A and C are correlated. Why is it possible for B and C to be uncorrelated?

Imagine a random point on a plane with coordinates $(x, y)$, where $x, y \in [-1, 1]$. A = both $x$ and $y$ are positive B = $x$ is positive C = $y$ is positive It is clear A is correlated with ...
Jozef Mikušinec's user avatar
14 votes

If A and B are correlated and A and C are correlated. Why is it possible for B and C to be uncorrelated?

EDIT I have a better simulation ...
Dave's user avatar
  • 3,960
11 votes
Accepted

Is there an asymmetric version of nominal correlation?

I found what I was looking for - it's called Theil's U, or the Uncertainty Coefficient. I've used it in this Kaggle kernel, you can check it out for an example and code implementation in Python EDIT:...
shakedzy's user avatar
  • 699
10 votes
Accepted

Is a correlation matrix meaningful for a binary classification task?

Well correlation, namely Pearson coefficient, is built for continuous data. Thus when applied to binary/categorical data, you will obtain measure of a relationship which does not have to be correct ...
HonzaB's user avatar
  • 1,669
10 votes

How does multicollinearity affect neural networks?

I just came across a research paper that answers this question. In case this helps anyone in the future, the paper Multicollinearity: A tale of two nonparametric regressions mentions that neural ...
user3667125's user avatar
8 votes

In supervised learning, why is it bad to have correlated features?

In perspective of storing data in databases, storing correlated features is somehow similar to storing redundant information which it may cause wasting of storage and also it may cause inconsistent ...
Green Falcon's user avatar
  • 14.1k
7 votes
Accepted

Why is pandas corr() deleting columns?

Pearson's correlation is the default correlation used with Pandas corr method. Categorical features ( not numerical ) are ignored during this process due to their nature of not being continuous. It ...
Blenz's user avatar
  • 2,084
7 votes
Accepted

Is autocorrelation of residuals a problem in machine learning?

Yes, autocorrelation in residuals is a problem, but this is essentially because it is a clear illustration that there was more learnable information in the process you are modelling but your model ...
Nicholas James Bailey's user avatar
6 votes
Accepted

How do i interpret this correlation

No. From this correlation matrix you cannot draw the conclusion that as long as the student has good gpa and good gre even though his Alma Mater's prestige is low - he will get admitted in a ...
Gino_JrDataScientist's user avatar
6 votes
Accepted

Generate predictions that are orthogonal (uncorrelated) to a given variable

This requirement can be satisfied by adding sufficient noise to predictions $\hat{y}$ to decorrelate them from orthogonal values $v$. Ideally, if $\hat{y}$ is already decorrelated from $v$, no noise ...
Esmailian's user avatar
  • 9,342
6 votes

If A and B are correlated and A and C are correlated. Why is it possible for B and C to be uncorrelated?

You can see it with a constructive technique: Let's say A and B are correlated A and C are correlated B and C is uncorrelated How is it possible for B and C to be uncorrelated when they are both ...
Jeffrey's user avatar
  • 181
6 votes

Is autocorrelation of residuals a problem in machine learning?

Choose model A, if autocorrelation is significant residuals="mistakes in predictions" should be completely random, i.e. follow White noise. Now if something is significantly autocorrelated ...
Noah Weber's user avatar
  • 5,699
5 votes

In supervised learning, why is it bad to have correlated features?

Sometimes correlated features -- and the duplication of information that provides -- does not hurt a predictive system. Consider an ensemble of decision trees, each of which considers a sample of rows ...
Dan Jarratt's user avatar
5 votes
Accepted

Generalization of Correlation Coefficient

I assume that when you speak of correlation coeficient, you have the Pearson linear correlation in mind. Indeed, there are other options. Two very popular ones are the rank correlations respectively ...
Perochkin's user avatar
  • 311
5 votes

Correlation vs Multicollinearity

I'll go through your questions one by one: What if we have to check the correlation between a continuous and categorical variable? One option is to use point biserial correlation. You can read more ...
Leevo's user avatar
  • 6,265
5 votes
Accepted

Several independent variables based on the same underlying data

For predictive power, in general, including both shouldn't be a problem. But there is a lot of nuance here. Foremost, if predictive power isn't all you care about: if you're making statistical ...
Ben Reiniger's user avatar
  • 11.9k
5 votes

Can features negatively correlated with the target be used?

Negative correlation is not the same as low correlation. If variables $x$ and $y$ have a correlation value of $c$, then $-x$ and $y$ will have a correlation of $-c$. When people talk about "low ...
Itamar Mushkin's user avatar
5 votes

Remove correlated features before or after splitting test and training set?

By principle in supervised ML any decision which affects the model should be made using only the training set, in order to avoid data leakage. Following this principle requires the training/test split ...
Erwan's user avatar
  • 25.5k
4 votes
Accepted

Feature Selection and PCA

PCA simply finds more compact ways of representing correlated data. PCA does not explicitly compact the data in order to better explain the target variable. In some cases, most of your inputs might be ...
Ryan Zotti's user avatar
  • 4,149
4 votes

Methods for Determining Possible Causation Between Two Time Series

I find two possible solutions: Granger Causality and Convergent Cross Mapping (CCM). Granger Causality is based on the t-test of lagged value of first variable with the second variable. The ...
William's user avatar
  • 296
4 votes

Does XGBoost handle multicollinearity by itself?

A remark on Sandeep's answer: Assuming 2 of your features are highly colinear (say equal 99% of time) Indeed only 1 feature is selected at each split, but for the next split, the xgb can select the ...
PSAfrance's user avatar
4 votes

Is there a way to measure correlation between two similar datasets?

I would take a look at Canonical correlation Analysis.
Robin's user avatar
  • 1,337
4 votes
Accepted

Is there a way to measure correlation between two similar datasets?

I see a lot of people post this similar question on StackExchange, and the truth is that there is no methodology to compare if data set A looks like set B. You can compare summary statistics, such as ...
Jon's user avatar
  • 481

Only top scored, non community-wiki answers of a minimum length are eligible