85
Two Categorical Variables
Checking if two categorical variables are independent can be done with Chi-Squared test of independence.
This is a typical Chi-Square test: if we assume that two variables are independent, then the values of the contingency table for these variables should be distributed uniformly. And then we check how far away from uniform the ...
37
Correlated features in general don't improve models (although it depends on the specifics of the problem like the number of variables and the degree of correlation), but they affect specific models in different ways and to varying extents:
For linear models (e.g., linear regression or logistic regression), multicolinearity can yield solutions that are ...
33
Decision trees are by nature immune to multi-collinearity. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models such as Logistic regression would use both the features.
Since boosted trees use individual decision trees, they also are unaffected by multi-collinearity. ...
16
(Assuming you are talking about supervised learning)
Correlated features will not always worsen your model, but they will not always improve it either.
There are three main reasons why you would remove correlated features:
Make the learning algorithm faster
Due to the curse of dimensionality, less features usually mean high improvement in terms of speed.
...
11
You've really got a classification problem on your hands, not a regression problem. Your target is not continuous, and Pearson correlation measures a relationship between continuous variables really. That's problematic enough to start.
Low correlation means there's no linear relationship; it doesn't mean there's no information in the feature that predicts ...
9
Word of warning from a former airline Revenue Management analyst: you might be barking up the wrong tree with this approach. Apologies for the wall of text that follows, but this data is a lot more complex and noisy than might appear at first glance, so wanted to provide a short description of how it's generated; forewarned is forearmed.
Airline fares have ...
9
I was curious about this and made a few tests.
I’ve trained a model on the diamonds dataset, and observed that the variable “x” is the most important to predict whether the price of a diamond is higher than a certain threshold.
Then, I’ve added multiple columns highly correlated to x, ran the same model, and observed the same values.
It seems that when ...
9
There is an answer from Tianqi Chen (2018).
This difference has an impact on a corner case in feature importance analysis: the correlated features.
Imagine two features perfectly correlated, feature A and feature B. For one specific tree, if the algorithm needs one of them, it will choose randomly (true in both boosting and Random Forests™).
...
8
I found what I was looking for - it's called Theil's U, or the Uncertainty Coefficient.
I've used it in this Kaggle kernel, you can check it out for an example and code implementation in Python
EDIT: I also have a blogpost about it.
6
Well correlation, namely Pearson coefficient, is built for continuous data. Thus when applied to binary/categorical data, you will obtain measure of a relationship which does not have to be correct and/or precise.
There are quite a few answers on stats exchange covering this topic - this or this for example.
6
In perspective of storing data in databases, storing correlated features is somehow similar to storing redundant information which it may cause wasting of storage and also it may cause inconsistent data after updating or editing tuples.
If we add so much correlated features to the model we may cause the model to consider unnecessary features and we may ...
6
No. From this correlation matrix you cannot draw the conclusion that
as long as the student has good gpa and good gre even though his Alma Mater's prestige is low - he will get admitted in a college
The reason is that correlation is a measure of association between single pairs of variables. The conclusion you draw above - on the contrary - is based on a ...
6
Pearson's correlation is the default correlation used with Pandas corr method.
Categorical features ( not numerical ) are ignored during this process due to their nature of not being continuous. It makes no sense to say if categorical_var1 is increased by one , categorical_var2 also increases by X ( X's value depends on the correlation between the 2 ...
5
Common rule in machine learning is to try simple things first. For predicting continuous variables there's nothing more basic than simple linear regression. "Simple" in the name means that there's only one predictor variable used (+ intercept, of course):
y = b0 + x*b1
where b0 is an intercept and b1 is a slope. For example, you may want to predict ...
5
Sometimes correlated features -- and the duplication of information that provides -- does not hurt a predictive system. Consider an ensemble of decision trees, each of which considers a sample of rows and a sample of columns. If two columns are highly correlated, there's a chance that one of them won't be selected in a particular tree's column sample, and ...
5
I assume that when you speak of correlation coeficient, you have the Pearson linear correlation in mind. Indeed, there are other options. Two very popular ones are the rank correlations respectively called Spearman's $\rho$ and Kendall's $\tau$.
To give you an idea of what they are, consider $n$ observations from a $d$-dimensional random vector $X = (X_1,\...
5
This requirement can be satisfied by adding sufficient noise to predictions $\hat{y}$ to decorrelate them from orthogonal values $v$. Ideally, if $\hat{y}$ is already decorrelated from $v$, no noise would be added to $\hat{y}$, thus $\hat{y}$ would be maximally correlated with $y$.
Mathematically, we want to create $\hat{y}'=\hat{y}+\epsilon$ from $\...
5
For predictive power, in general, including both shouldn't be a problem. But there is a lot of nuance here.
Foremost, if predictive power isn't all you care about: if you're making statistical inferences, or care about explainability and feature importances, then including both can cause issues. Briefly, your model may split the importance of the underlying ...
4
In addition to exploratory data analysis (EDA), both descriptive and visual, I would try to use time series analysis as a more comprehensive and sophisticated analysis. Specifically, I would perform time series regression analysis. Time series analysis is a huge research and practice domain, so, if you're not familiar with the fundamentals, I suggest ...
4
The idea you have in mind is called "feature selection" or "attribute selection". The fact that you have a categorical dependent variable and continuous independent variables is mostly irrelevant because you're expected to use an algorithm or statistical method that is suitable for your requirements.
As for feature selection methods, there are several ...
4
If the correlation between two features $x_1$ and $x_2$ is 1 that means that you can write $x_1 = c\cdot x_2 + a$. The only knew knowledge there is are those two constants, the individual values can be retrieved knowing this. I highly doubt there is anything a machine learning algorithm can learn from this and it is a fact that for some having this kind of ...
4
I find two possible solutions: Granger Causality and Convergent Cross Mapping (CCM).
Granger Causality is based on the t-test of lagged value of first variable with the second variable. The limitation is that it could not eliminate the confounding effect.
The CCM first build the manifold for the first variable X. Then for each time point, weights of ...
4
I see a lot of people post this similar question on StackExchange, and the truth is that there is no methodology to compare if data set A looks like set B. You can compare summary statistics, such as means, deviations, min/max, but there's no magical formula to say that data set A looks like B, especially if they are varying data sets by rows and columns.
I ...
4
Having highly correlated features is a type of redundancy in features. And yes, it effects a regression model if you are having highly correlated features. A very nice explanation is given here.
PCA is a nice choice when it comes to dimensionality reduction.
4
Why is Multicollinearity a Potential Problem?
A key goal of regression analysis is to isolate the relationship between each independent variable and the dependent variable. The interpretation of a regression coefficient is that it represents the mean change in the dependent variable for each 1 unit change in an independent variable when you hold all of the ...
4
as both components need to be equally weighted in order for the feature to make sense
That is not the case.
For instance if $\text{sin}(\theta)$ of the cyclical feature is weighted strongly, it means that the original feature has the strongest positive effect on output at $\theta = \frac{\pi}{2}$.
If the two features are weighted equally, then the focus ...
4
An alternative to the one provided by @Kasra is dimensionality reduction. It's another way of solving your multicollinearity problems, while avoiding deleting variables more or less arbitrarily.
You can use simpler, linear techniques such as PCA, or more complex non-linear techniques such as Autoencoders. t-SNE is a non-linear technique that is typically ...
Only top voted, non community-wiki answers of a minimum length are eligible
