93
Two Categorical Variables
Checking if two categorical variables are independent can be done with Chi-Squared test of independence.
This is a typical Chi-Square test: if we assume that two variables are independent, then the values of the contingency table for these variables should be distributed uniformly. And then we check how far away from uniform the ...
47
Correlated features in general don't improve models (although it depends on the specifics of the problem like the number of variables and the degree of correlation), but they affect specific models in different ways and to varying extents:
For linear models (e.g., linear regression or logistic regression), multicolinearity can yield solutions that are ...
35
Decision trees are by nature immune to multi-collinearity. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models such as Logistic regression would use both the features.
Since boosted trees use individual decision trees, they also are unaffected by multi-collinearity. ...
22
(Assuming you are talking about supervised learning)
Correlated features will not always worsen your model, but they will not always improve it either.
There are three main reasons why you would remove correlated features:
Make the learning algorithm faster
Due to the curse of dimensionality, less features usually mean high improvement in terms of speed.
...
15
Imagine a random point on a plane with coordinates $(x, y)$, where $x, y \in [-1, 1]$.
A = both $x$ and $y$ are positive
B = $x$ is positive
C = $y$ is positive
It is clear A is correlated with both B and C, which are not themselves correlated (assuming uniform distribution).
12
I was curious about this and made a few tests.
I’ve trained a model on the diamonds dataset, and observed that the variable “x” is the most important to predict whether the price of a diamond is higher than a certain threshold.
Then, I’ve added multiple columns highly correlated to x, ran the same model, and observed the same values.
It seems that when ...
12
There is an answer from Tianqi Chen (2018).
This difference has an impact on a corner case in feature importance analysis: the correlated features.
Imagine two features perfectly correlated, feature A and feature B. For one specific tree, if the algorithm needs one of them, it will choose randomly (true in both boosting and Random Forests™).
...
12
EDIT
I have a better simulation
set.seed(2020)
N <- 250
X1 <- rnorm(N, 0, 1)
X2 <- rnorm(N, 0, 1)
X3 <- X1 + X2
par(mfrow=c(3,1))
plot(X1, X3)
plot(X2, X3)
plot(X1, X2)
cor.test(X1, X3) # 95% confidence interval: [0.6719684, 0.7870920]
cor.test(X2, X3) # 95% confidence interval: [0.5767864, 0.7197146]
cor.test(X1, X2) # 95% confidence interval: ...
11
You've really got a classification problem on your hands, not a regression problem. Your target is not continuous, and Pearson correlation measures a relationship between continuous variables really. That's problematic enough to start.
Low correlation means there's no linear relationship; it doesn't mean there's no information in the feature that predicts ...
9
Word of warning from a former airline Revenue Management analyst: you might be barking up the wrong tree with this approach. Apologies for the wall of text that follows, but this data is a lot more complex and noisy than might appear at first glance, so wanted to provide a short description of how it's generated; forewarned is forearmed.
Airline fares have ...
8
Why is Multicollinearity a Potential Problem?
A key goal of regression analysis is to isolate the relationship between each independent variable and the dependent variable. The interpretation of a regression coefficient is that it represents the mean change in the dependent variable for each 1 unit change in an independent variable when you hold all of the ...
8
I found what I was looking for - it's called Theil's U, or the Uncertainty Coefficient.
I've used it in this Kaggle kernel, you can check it out for an example and code implementation in Python
EDIT: I also have a blogpost about it.
7
Well correlation, namely Pearson coefficient, is built for continuous data. Thus when applied to binary/categorical data, you will obtain measure of a relationship which does not have to be correct and/or precise.
There are quite a few answers on stats exchange covering this topic - this or this for example.
7
In perspective of storing data in databases, storing correlated features is somehow similar to storing redundant information which it may cause wasting of storage and also it may cause inconsistent data after updating or editing tuples.
If we add so much correlated features to the model we may cause the model to consider unnecessary features and we may ...
7
Yes, autocorrelation in residuals is a problem, but this is essentially because it is a clear illustration that there was more learnable information in the process you are modelling but your model missed it.
In the unlikely event that you have two equally performant models but one shows significant autocorrelation (you can test for this using the Durbin-...
6
Sometimes correlated features -- and the duplication of information that provides -- does not hurt a predictive system. Consider an ensemble of decision trees, each of which considers a sample of rows and a sample of columns. If two columns are highly correlated, there's a chance that one of them won't be selected in a particular tree's column sample, and ...
6
No. From this correlation matrix you cannot draw the conclusion that
as long as the student has good gpa and good gre even though his Alma Mater's prestige is low - he will get admitted in a college
The reason is that correlation is a measure of association between single pairs of variables. The conclusion you draw above - on the contrary - is based on a ...
6
Pearson's correlation is the default correlation used with Pandas corr method.
Categorical features ( not numerical ) are ignored during this process due to their nature of not being continuous. It makes no sense to say if categorical_var1 is increased by one , categorical_var2 also increases by X ( X's value depends on the correlation between the 2 ...
6
You can see it with a constructive technique:
Let's say A and B are correlated A and C are correlated B and C is uncorrelated How is it possible for B and C to be uncorrelated when they are both correlated to A?
Pick B from a random distribution. Dice throws, random values between 1 and 6.
Pick C from a random distribution. Another set of different dice ...
6
Choose model A, if autocorrelation is significant
residuals="mistakes in predictions" should be completely random, i.e. follow White noise. Now if something is significantly autocorrelated it wont be truly random and the independent error model is incorrect and it wont be a robust variance estimator. Prefer model A
How to measure significant ...
5
Common rule in machine learning is to try simple things first. For predicting continuous variables there's nothing more basic than simple linear regression. "Simple" in the name means that there's only one predictor variable used (+ intercept, of course):
y = b0 + x*b1
where b0 is an intercept and b1 is a slope. For example, you may want to predict ...
5
I assume that when you speak of correlation coeficient, you have the Pearson linear correlation in mind. Indeed, there are other options. Two very popular ones are the rank correlations respectively called Spearman's $\rho$ and Kendall's $\tau$.
To give you an idea of what they are, consider $n$ observations from a $d$-dimensional random vector $X = (X_1,\...
5
This requirement can be satisfied by adding sufficient noise to predictions $\hat{y}$ to decorrelate them from orthogonal values $v$. Ideally, if $\hat{y}$ is already decorrelated from $v$, no noise would be added to $\hat{y}$, thus $\hat{y}$ would be maximally correlated with $y$.
Mathematically, we want to create $\hat{y}'=\hat{y}+\epsilon$ from $\...
5
For predictive power, in general, including both shouldn't be a problem. But there is a lot of nuance here.
Foremost, if predictive power isn't all you care about: if you're making statistical inferences, or care about explainability and feature importances, then including both can cause issues. Briefly, your model may split the importance of the underlying ...
4
In addition to exploratory data analysis (EDA), both descriptive and visual, I would try to use time series analysis as a more comprehensive and sophisticated analysis. Specifically, I would perform time series regression analysis. Time series analysis is a huge research and practice domain, so, if you're not familiar with the fundamentals, I suggest ...
4
The idea you have in mind is called "feature selection" or "attribute selection". The fact that you have a categorical dependent variable and continuous independent variables is mostly irrelevant because you're expected to use an algorithm or statistical method that is suitable for your requirements.
As for feature selection methods, there are several ...
4
If the correlation between two features $x_1$ and $x_2$ is 1 that means that you can write $x_1 = c\cdot x_2 + a$. The only knew knowledge there is are those two constants, the individual values can be retrieved knowing this. I highly doubt there is anything a machine learning algorithm can learn from this and it is a fact that for some having this kind of ...
4
PCA simply finds more compact ways of representing correlated data. PCA does not explicitly compact the data in order to better explain the target variable. In some cases, most of your inputs might be correlated with each other but have minimal relevance to your target variable. That's probably what is happening in your case.
Consider a toy example. Lets ...
Only top voted, non community-wiki answers of a minimum length are eligible
