# Questions tagged [correlation]

A measure of the degree of linear association among a pair of variables.

401 questions
Filter by
Sorted by
Tagged with
381k views

### How to get correlation between two categorical variable and a categorical variable and continuous variable?

I am building a regression model and I need to calculate the below to check for correlations Correlation between 2 Multi level categorical variables Correlation between a Multi level categorical ...
• 2,058
123k views

### In supervised learning, why is it bad to have correlated features?

I read somewhere that if we have features that are too correlated, we have to remove one, as this may worsen the model. It is clear that correlated features means that they bring the same information, ...
• 1,289
62k views

### Does XGBoost handle multicollinearity by itself?

I'm currently using XGBoost on a data-set with 21 features (selected from list of some 150 features), then one-hot coded them to obtain ~98 features. A few of these 98 features are somewhat redundant, ...
• 1,783
50k views

### Pearson vs Spearman vs Kendall

What are the characteristics of the three correlation coefficients and what are the comparisons of each of them/assumptions? Can somebody kindly take me through the concepts?
15k views

### How does multicollinearity affect neural networks?

Multicollinearity is a problem for linear regression because the results become unstable / depend too much on single elements (source). (Also, the inverse of $X^TX$ doesn't exist so the standard OLS ...
• 19k
314 views

### Recognize a grammar in a sequence of fuzzy tokens

I have text documents which contain mainly lists of Items. Each Item is a group of several token from different types: FirstName, LastName, BirthDate, PhoneNumber, City, Occupation, etc. A token is a ...
• 241
4k views

### If A and B are correlated and A and C are correlated. Why is it possible for B and C to be uncorrelated?

Let's say A and B are correlated A and C are correlated B and C is uncorrelated How is it possible for B and C to be uncorrelated when they are both correlated to A?
• 131
2k views

### Airline Fares - What analysis should be used to detect competitive price-setting behavior and price correlations?

I want to investigate price-setting behavior of airlines -- specifically how airlines react to competitors pricing. As I would say my knowledge about more complex analysis is quite limited I've done ...
• 221
15k views

### Dissmissing features based on correlation with target variable

Is it valid to dismiss features based on their Pearson correlation values with the target variable in a classification problem? say for instance I have a dataset with the following format where the ...
• 275
352 views

### Generate predictions that are orthogonal (uncorrelated) to a given variable

I have an X matrix, a y variable, and another variable ORTHO_VAR. I need to predict the <...
• 224
223 views

### Features reduction for the not correlated data set

I am working with classification problem on a training data set, which have 100 features. All the features in pairs haven't visible correlation. One can see it in the example pair plot for the some of ...
4k views

### How to find similarity between different factors in a dataset

Introduction Let's say I have a dataset of different observation of different people and I want to group people together to know which person is closest to the other one. I also want to have a ...
• 183
1k views

### What should be the value of non-rated field when finding cosine similarity

I am working on a very basic book recommender system. I want to know what to do with the fields which aren't rated by the user when finding cosine similarity, should we ignore them and calculate only ...
• 181
11k views

### Is there a way to measure correlation between two similar datasets?

Let's say that I have two similar datasets with the same size of elements, for example 3D points : Dataset A : { (1,2,3), (2,3,4), (4,2,1) } Dataset B : { (2,1,3), (2,4,6), (8,2,3) } And the ...
• 233
2k views

### which neural network topology to learn correlations between time series?

I have two (or more in principle) 1xN time series, and I would like to train a NN to predict the next value of both. I can arrange them as a 2xN matrix and feed a window from this matrix as input to ...
• 221
33k views

### Is a correlation matrix meaningful for a binary classification task?

When examining my dataset with a binary target (y) variable I wonder if a correlation matrix is useful to determine predictive power of each variable. My predictors (X) contain some numeric and some ...
• 337
2k views

### Correlation vs Multicollinearity

I have been taught to check correlation matrix before going for any algorithm. I have a few questions around the same: Pearson Correlation is for numerical variables only. What if we have to check ...
• 161
4k views

### Why is a correlation matrix symmetric?

I'm sorry for being so weak in math. (I'm a student) For eg. this is a correlation matrix. ...
3k views

### How to more simply see overlapping data for dozens of variables?

I'm trying to think of the best way to see how multiple variables (about 40) related to a very large userbase can be seen to interact with one another. As an analogy, imagine I had survey data of ...
• 113
92 views

### Fitting model to differenced time series

I have a time series on daily stock price of company(2013 data points).I took a first order difference and the following acf and pacf plots of the differenced series were obtained. However, I am ...
• 231
790 views

### What can be done with highly correlated variables (>.95 and <-.95)

I hope we can remove the highly correlated variables based on the feature importance may be with PCA etc. Is there anything we can do with highly correlated variables/ Thanks in advance !
• 309
1k views

### Determine highly correlated segments

Given a dataset that has a binary (0/1) dependent variable and a large collection of continuous and categorical independent variables, is there a process and ideally a R package that can find ...
• 151
1k views

### What is the best Data Mining algorithm for prediction based on a single variable?

I have a variable whose value I would like to predict, and I would like to use only one variable as predictor. For instance, predict traffic density based on weather. Initially, I thought about using ...
• 420
4k views

### SVD for recommendation engine

I'm trying to build a toy recommendation engine to wrap my mind around Singular Value Decomposition (SVD). I've read enough content to understand the motivations and intuition behind the actual ...
2k views

### How to identify recurring patterns in this set of transactional data

I'm working on a dataset of banking transactions and would like to find recurrent transactions. I've been mapping transactions per merchant in timeseries, and tried to use acf from statsmodels.tsa....
106 views

### Detecting abundance of a certain periodic pattern in a time series?

I am really stumped at the moment about how to solve a particular problem. I have many time series like this: This represents the number of hours a person spends on a website each day throughout the ...
160 views

### Influence of trend on (supposedly) correlated time series

TL;DR: What is the impact of a linear trend on the correlation between time series that are (most likely) not spuriously correlated? I'm currently trying to reconstruct/cross-validate an analysis ...
• 153
2k views

### Converting non-numeric data values into equivalent rank scores

Consider a data-frame similar to the one shown (the actual data-frame is much larger) ...
• 1,783
165 views

### How to interpret two continous variables output using GAM?

I really need help with GAM. I have to find out whether association is linear or non-linear by using GAM. The predictor variable is temperature at lag0 and the output is cardiovascular admissions (...
1k views

### Is autocorrelation of residuals a problem in machine learning?

Let's assume I have a random forest model and the residuals of the model are autocorrelated. Is this a problem? As an example, let's assume I have two different random forest models, A and B, with a ...
• 595
2k views

### Is there an asymmetric version of nominal correlation?

I use Cramer's V to calculate correlation of features in a dataset made of only nominal features. Let's consider the following dataset: ...
• 699
4k views

### Why is pandas corr() deleting columns?

I'm doing a basic correlation analysis but for some reason pandas corr() is deleting columns, not sure why. ...
• 87
3k views

### is it better to correlate and encode or encode and correlate?

I have one doubt like is it better to perform label encoding and check for the correlation or should I 1st perform correlation and do label encoding? Because when I tried it both ways I'm getting ...
• 181
3k views

### Chose among highly correlated variables

I am working on a Kaggle dataset and I am trying to build a predictive model for the "Chance of Admit" (dependent variable) of students to the university of their interest. Below you can find the ...
• 149
2k views

### Is Pearson coefficient a good indicator of dependency between variables?

Once I have been asked how would I calculate correlation between two time series. Since I am new to data science I answered: "I would just calculate the Pearson correlation coefficient". That wasn't a ...
• 343
621 views

### Generalization of Correlation Coefficient

The correlation coefficient tells me how two variables (sequences of numbers) are correlated with each other. Does it generalize to non-linear scenarios? How could one more generally measure the ...
398 views

### Methods for Determining Possible Causation Between Two Time Series

I'll set this question up as a simplified example: We hypothesize that there is a causal relationship between average gasoline prices and road traffic in a particular city. The data cover the same ...
1k views

### Can i expect good results having low correlation attributes?

This was a question i saw in an interview for a data scientist position: "Here is the following correlation heatmap that i got from my attributes. Regarding the correlation of each feature with ...
• 395
275 views

### Should Feature Selection processes be apply on training data or on all data?

I've realized that on examples and guides, sometimes feature selection processes (correlation elimination, backward/stepwise) are applied on the train data after splitting all data but on the other ...
• 173
2k views

### Hive: How to calculate the Kendall coefficient of correlation of a pair of a numeric columns in the group?

In this wiki page there is a function corr() that calculates the Pearson coefficient of correlation, but my question is that: is there any function in Hive that ...
• 235
451 views

### How to make sure that the features learned by a neural network are not correlated?

Each layer of a neural network learns features of the input data. The first layer learns low-level features (e.g. edges in images). Each subsequent layer learns more abstract features. Then the ...
• 1,146
704 views

### Correlation between examples

I'm training a Neural Network for pattern recognition. I have a matrix of examples of size ($N$x$4$) with $N$ examples and $4$ variables. When I train the Network the Number of examples used for ...
• 63
6k views

### Is $R^2$ an appropriate evaluation metric for k-Nearest Neighbors?

I found a source that stated that $R^2$ is the ”percentage of the response variable variation that is explained by a linear model.” (Source) Since kNN is not a linear model (it is nonparametric), is ...
• 293
360 views

### Minimize correlation between input and output of black box system

I am not sure if "minimize correlation" is the right title for this issue but I could not find a better sentence to describe what I would like to achieve. Let's say that I have a black box with ...
• 141
571 views

### what is the difference in terms namely Correlation, correlated and collinearity?

A website says Correlation refers to an increase/decrease in a dependent variable with an increase/decrease in an independent variable. Collinearity refers to two or more independent variables acting ...
568 views

### Dealing with correlated features when calculating permutation importance

I have implemented the permutation importance calculation as found here in my attempt to identify features that contribute little to my model's (Gradient Boosted Tree model) predictive power. The ...
• 182
555 views

### Machine learning algorithm that uses the Pearson or Spearman correlation?

I've come across linear and multiple regression, SVM, random forests. Does any know of a machine learning algorithm that uses the Pearson correlation or Spearman correlation? Best, Dave
1k views

### SelectKBest and Correlation returns me excatly same feature selection. How?

Im working on selecting most effective features from a dataset with over that 2000 features. Im using different algorithms for that (selectKBest with chi-square, Extra Trees, Correlation etc.) But ...
6k views

### Can features negatively correlated with the target be used?

In feature selection (for a regression problem), can features that are negatively correlated with the target variable be chosen to predict the target? I don't think negative correlation means the ...
• 277