71
votes
Accepted
In supervised learning, why is it bad to have correlated features?
Correlated features in general don't improve models (although it depends on the specifics of the problem like the number of variables and the degree of correlation), but they affect specific models in ...
- 1,167
54
votes
Accepted
Does XGBoost handle multicollinearity by itself?
Decision trees are by nature immune to multi-collinearity. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models ...
- 2,397
45
votes
Accepted
Pearson vs Spearman vs Kendall
Correlation is a bivariate analysis that measures the strength of association between two variables and the direction of the relationship. In terms of the strength of the relationship, the value of ...
- 2,624
36
votes
In supervised learning, why is it bad to have correlated features?
(Assuming you are talking about supervised learning)
Correlated features will not always worsen your model, but they will not always improve it either.
There are three main reasons why you would ...
- 5,226
23
votes
Does XGBoost handle multicollinearity by itself?
I was curious about this and made a few tests.
I’ve trained a model on the diamonds dataset, and observed that the variable “x” is the most important to predict whether the price of a diamond is ...
- 331
22
votes
Does XGBoost handle multicollinearity by itself?
There is an answer from Tianqi Chen (2018).
This difference has an impact on a corner case in feature importance analysis: the correlated features.
Imagine two features perfectly correlated, ...
- 340
17
votes
Accepted
Dissmissing features based on correlation with target variable
You've really got a classification problem on your hands, not a regression problem. Your target is not continuous, and Pearson correlation measures a relationship between continuous variables really. ...
- 6,475
16
votes
If A and B are correlated and A and C are correlated. Why is it possible for B and C to be uncorrelated?
Imagine a random point on a plane with coordinates $(x, y)$, where $x, y \in [-1, 1]$.
A = both $x$ and $y$ are positive
B = $x$ is positive
C = $y$ is positive
It is clear A is correlated with ...
- 261
15
votes
In supervised learning, why is it bad to have correlated features?
Why is Multicollinearity a Potential Problem?
A key goal of regression analysis is to isolate the relationship between each independent variable and the dependent variable. The interpretation of a ...
- 151
13
votes
If A and B are correlated and A and C are correlated. Why is it possible for B and C to be uncorrelated?
EDIT
I have a better simulation
...
- 3,601
11
votes
Accepted
Is there an asymmetric version of nominal correlation?
I found what I was looking for - it's called Theil's U, or the Uncertainty Coefficient.
I've used it in this Kaggle kernel, you can check it out for an example and code implementation in Python
EDIT:...
- 679
10
votes
Accepted
Is a correlation matrix meaningful for a binary classification task?
Well correlation, namely Pearson coefficient, is built for continuous data. Thus when applied to binary/categorical data, you will obtain measure of a relationship which does not have to be correct ...
- 1,671
9
votes
Airline Fares - What analysis should be used to detect competitive price-setting behavior and price correlations?
Word of warning from a former airline Revenue Management analyst: you might be barking up the wrong tree with this approach. Apologies for the wall of text that follows, but this data is a lot more ...
- 191
8
votes
In supervised learning, why is it bad to have correlated features?
In perspective of storing data in databases, storing correlated features is somehow similar to storing redundant information which it may cause wasting of storage and also it may cause inconsistent ...
- 13.5k
7
votes
How does multicollinearity affect neural networks?
I just came across a research paper that answers this question. In case this helps anyone in the future, the paper Multicollinearity: A tale of two nonparametric regressions mentions that neural ...
- 171
7
votes
Accepted
Why is pandas corr() deleting columns?
Pearson's correlation is the default correlation used with Pandas corr method.
Categorical features ( not numerical ) are ignored during this process due to their nature of not being continuous. It ...
- 2,024
7
votes
Accepted
Is autocorrelation of residuals a problem in machine learning?
Yes, autocorrelation in residuals is a problem, but this is essentially because it is a clear illustration that there was more learnable information in the process you are modelling but your model ...
- 1,480
6
votes
Accepted
How do i interpret this correlation
No. From this correlation matrix you cannot draw the conclusion that
as long as the student has good gpa and good gre even though his Alma Mater's prestige is low - he will get admitted in a ...
6
votes
Accepted
Generate predictions that are orthogonal (uncorrelated) to a given variable
This requirement can be satisfied by adding sufficient noise to predictions $\hat{y}$ to decorrelate them from orthogonal values $v$. Ideally, if $\hat{y}$ is already decorrelated from $v$, no noise ...
- 8,767
6
votes
If A and B are correlated and A and C are correlated. Why is it possible for B and C to be uncorrelated?
You can see it with a constructive technique:
Let's say A and B are correlated A and C are correlated B and C is uncorrelated How is it possible for B and C to be uncorrelated when they are both ...
- 181
6
votes
Is autocorrelation of residuals a problem in machine learning?
Choose model A, if autocorrelation is significant
residuals="mistakes in predictions" should be completely random, i.e. follow White noise. Now if something is significantly autocorrelated ...
- 5,409
5
votes
Accepted
What is the best Data Mining algorithm for prediction based on a single variable?
Common rule in machine learning is to try simple things first. For predicting continuous variables there's nothing more basic than simple linear regression. "Simple" in the name means that there's ...
- 2,791
5
votes
In supervised learning, why is it bad to have correlated features?
Sometimes correlated features -- and the duplication of information that provides -- does not hurt a predictive system. Consider an ensemble of decision trees, each of which considers a sample of rows ...
- 295
5
votes
Accepted
Generalization of Correlation Coefficient
I assume that when you speak of correlation coeficient, you have the Pearson linear correlation in mind. Indeed, there are other options. Two very popular ones are the rank correlations respectively ...
- 311
5
votes
Correlation vs Multicollinearity
I'll go through your questions one by one:
What if we have to check the correlation between a continuous and
categorical variable?
One option is to use point biserial correlation. You can read more ...
- 5,747
5
votes
Accepted
Several independent variables based on the same underlying data
For predictive power, in general, including both shouldn't be a problem. But there is a lot of nuance here.
Foremost, if predictive power isn't all you care about: if you're making statistical ...
- 10.5k
4
votes
Airline Fares - What analysis should be used to detect competitive price-setting behavior and price correlations?
In addition to exploratory data analysis (EDA), both descriptive and visual, I would try to use time series analysis as a more comprehensive and sophisticated analysis. Specifically, I would perform ...
- 6,518
4
votes
Determine highly correlated segments
The idea you have in mind is called "feature selection" or "attribute selection". The fact that you have a categorical dependent variable and continuous independent variables is mostly irrelevant ...
- 858
Only top scored, non community-wiki answers of a minimum length are eligible