Skip to main content

Questions tagged [correlation]

A measure of the degree of linear association among a pair of variables.

Filter by
Sorted by
Tagged with
132 votes
1 answer
381k views

How to get correlation between two categorical variable and a categorical variable and continuous variable?

I am building a regression model and I need to calculate the below to check for correlations Correlation between 2 Multi level categorical variables Correlation between a Multi level categorical ...
GeorgeOfTheRF's user avatar
91 votes
7 answers
123k views

In supervised learning, why is it bad to have correlated features?

I read somewhere that if we have features that are too correlated, we have to remove one, as this may worsen the model. It is clear that correlated features means that they bring the same information, ...
Spider's user avatar
  • 1,289
59 votes
6 answers
62k views

Does XGBoost handle multicollinearity by itself?

I'm currently using XGBoost on a data-set with 21 features (selected from list of some 150 features), then one-hot coded them to obtain ~98 features. A few of these 98 features are somewhat redundant, ...
neural-nut's user avatar
  • 1,783
41 votes
1 answer
50k views

Pearson vs Spearman vs Kendall

What are the characteristics of the three correlation coefficients and what are the comparisons of each of them/assumptions? Can somebody kindly take me through the concepts?
user avatar
22 votes
3 answers
15k views

How does multicollinearity affect neural networks?

Multicollinearity is a problem for linear regression because the results become unstable / depend too much on single elements (source). (Also, the inverse of $X^TX$ doesn't exist so the standard OLS ...
Martin Thoma's user avatar
14 votes
1 answer
314 views

Recognize a grammar in a sequence of fuzzy tokens

I have text documents which contain mainly lists of Items. Each Item is a group of several token from different types: FirstName, LastName, BirthDate, PhoneNumber, City, Occupation, etc. A token is a ...
OoDeLally's user avatar
  • 241
13 votes
8 answers
4k views

If A and B are correlated and A and C are correlated. Why is it possible for B and C to be uncorrelated?

Let's say A and B are correlated A and C are correlated B and C is uncorrelated How is it possible for B and C to be uncorrelated when they are both correlated to A?
Ashley's user avatar
  • 131
12 votes
3 answers
2k views

Airline Fares - What analysis should be used to detect competitive price-setting behavior and price correlations?

I want to investigate price-setting behavior of airlines -- specifically how airlines react to competitors pricing. As I would say my knowledge about more complex analysis is quite limited I've done ...
s1x's user avatar
  • 221
11 votes
2 answers
15k views

Dissmissing features based on correlation with target variable

Is it valid to dismiss features based on their Pearson correlation values with the target variable in a classification problem? say for instance I have a dataset with the following format where the ...
MedAli's user avatar
  • 275
9 votes
1 answer
352 views

Generate predictions that are orthogonal (uncorrelated) to a given variable

I have an X matrix, a y variable, and another variable ORTHO_VAR. I need to predict the <...
Chris's user avatar
  • 224
9 votes
2 answers
223 views

Features reduction for the not correlated data set

I am working with classification problem on a training data set, which have 100 features. All the features in pairs haven't visible correlation. One can see it in the example pair plot for the some of ...
Ruben Kazumov's user avatar
8 votes
3 answers
4k views

How to find similarity between different factors in a dataset

Introduction Let's say I have a dataset of different observation of different people and I want to group people together to know which person is closest to the other one. I also want to have a ...
zipp's user avatar
  • 183
8 votes
2 answers
1k views

What should be the value of non-rated field when finding cosine similarity

I am working on a very basic book recommender system. I want to know what to do with the fields which aren't rated by the user when finding cosine similarity, should we ignore them and calculate only ...
divyum's user avatar
  • 181
7 votes
5 answers
11k views

Is there a way to measure correlation between two similar datasets?

Let's say that I have two similar datasets with the same size of elements, for example 3D points : Dataset A : { (1,2,3), (2,3,4), (4,2,1) } Dataset B : { (2,1,3), (2,4,6), (8,2,3) } And the ...
xtluo's user avatar
  • 233
7 votes
3 answers
2k views

which neural network topology to learn correlations between time series?

I have two (or more in principle) 1xN time series, and I would like to train a NN to predict the next value of both. I can arrange them as a 2xN matrix and feed a window from this matrix as input to ...
Ziofil's user avatar
  • 221
6 votes
2 answers
33k views

Is a correlation matrix meaningful for a binary classification task?

When examining my dataset with a binary target (y) variable I wonder if a correlation matrix is useful to determine predictive power of each variable. My predictors (X) contain some numeric and some ...
Georg Heiler's user avatar
6 votes
3 answers
2k views

Correlation vs Multicollinearity

I have been taught to check correlation matrix before going for any algorithm. I have a few questions around the same: Pearson Correlation is for numerical variables only. What if we have to check ...
Payal Bhatia's user avatar
6 votes
3 answers
4k views

Why is a correlation matrix symmetric?

I'm sorry for being so weak in math. (I'm a student) For eg. this is a correlation matrix. ...
LazyAwkwardVaish Ok's user avatar
6 votes
2 answers
3k views

How to more simply see overlapping data for dozens of variables?

I'm trying to think of the best way to see how multiple variables (about 40) related to a very large userbase can be seen to interact with one another. As an analogy, imagine I had survey data of ...
Benny Lewis's user avatar
6 votes
0 answers
92 views

Fitting model to differenced time series

I have a time series on daily stock price of company(2013 data points).I took a first order difference and the following acf and pacf plots of the differenced series were obtained. However, I am ...
Jor_El's user avatar
  • 231
5 votes
3 answers
790 views

What can be done with highly correlated variables (>.95 and <-.95)

I hope we can remove the highly correlated variables based on the feature importance may be with PCA etc. Is there anything we can do with highly correlated variables/ Thanks in advance !
Wickkiey's user avatar
  • 309
5 votes
4 answers
1k views

Determine highly correlated segments

Given a dataset that has a binary (0/1) dependent variable and a large collection of continuous and categorical independent variables, is there a process and ideally a R package that can find ...
Bryan's user avatar
  • 151
5 votes
2 answers
1k views

What is the best Data Mining algorithm for prediction based on a single variable?

I have a variable whose value I would like to predict, and I would like to use only one variable as predictor. For instance, predict traffic density based on weather. Initially, I thought about using ...
doublebyte's user avatar
5 votes
2 answers
4k views

SVD for recommendation engine

I'm trying to build a toy recommendation engine to wrap my mind around Singular Value Decomposition (SVD). I've read enough content to understand the motivations and intuition behind the actual ...
William Gottschalk's user avatar
5 votes
2 answers
2k views

How to identify recurring patterns in this set of transactional data

I'm working on a dataset of banking transactions and would like to find recurrent transactions. I've been mapping transactions per merchant in timeseries, and tried to use acf from statsmodels.tsa....
loicbertron's user avatar
5 votes
2 answers
106 views

Detecting abundance of a certain periodic pattern in a time series?

I am really stumped at the moment about how to solve a particular problem. I have many time series like this: This represents the number of hours a person spends on a website each day throughout the ...
user avatar
5 votes
1 answer
160 views

Influence of trend on (supposedly) correlated time series

TL;DR: What is the impact of a linear trend on the correlation between time series that are (most likely) not spuriously correlated? I'm currently trying to reconstruct/cross-validate an analysis ...
Viktor Katzy's user avatar
5 votes
2 answers
2k views

Converting non-numeric data values into equivalent rank scores

Consider a data-frame similar to the one shown (the actual data-frame is much larger) ...
neural-nut's user avatar
  • 1,783
5 votes
2 answers
165 views

How to interpret two continous variables output using GAM?

I really need help with GAM. I have to find out whether association is linear or non-linear by using GAM. The predictor variable is temperature at lag0 and the output is cardiovascular admissions (...
Hasan Sohail's user avatar
4 votes
3 answers
1k views

Is autocorrelation of residuals a problem in machine learning?

Let's assume I have a random forest model and the residuals of the model are autocorrelated. Is this a problem? As an example, let's assume I have two different random forest models, A and B, with a ...
Funkwecker's user avatar
4 votes
1 answer
2k views

Is there an asymmetric version of nominal correlation?

I use Cramer's V to calculate correlation of features in a dataset made of only nominal features. Let's consider the following dataset: ...
shakedzy's user avatar
  • 699
4 votes
1 answer
4k views

Why is pandas corr() deleting columns?

I'm doing a basic correlation analysis but for some reason pandas corr() is deleting columns, not sure why. ...
raulb1's user avatar
  • 87
4 votes
2 answers
3k views

is it better to correlate and encode or encode and correlate?

I have one doubt like is it better to perform label encoding and check for the correlation or should I 1st perform correlation and do label encoding? Because when I tried it both ways I'm getting ...
Nithin Reddy's user avatar
4 votes
2 answers
3k views

Chose among highly correlated variables

I am working on a Kaggle dataset and I am trying to build a predictive model for the "Chance of Admit" (dependent variable) of students to the university of their interest. Below you can find the ...
batman's user avatar
  • 149
4 votes
3 answers
2k views

Is Pearson coefficient a good indicator of dependency between variables?

Once I have been asked how would I calculate correlation between two time series. Since I am new to data science I answered: "I would just calculate the Pearson correlation coefficient". That wasn't a ...
WoofDoggy's user avatar
  • 343
4 votes
1 answer
621 views

Generalization of Correlation Coefficient

The correlation coefficient tells me how two variables (sequences of numbers) are correlated with each other. Does it generalize to non-linear scenarios? How could one more generally measure the ...
Semihcan Doken's user avatar
4 votes
1 answer
398 views

Methods for Determining Possible Causation Between Two Time Series

I'll set this question up as a simplified example: We hypothesize that there is a causal relationship between average gasoline prices and road traffic in a particular city. The data cover the same ...
Thomas Kiehne's user avatar
4 votes
2 answers
1k views

Can i expect good results having low correlation attributes?

This was a question i saw in an interview for a data scientist position: "Here is the following correlation heatmap that i got from my attributes. Regarding the correlation of each feature with ...
heresthebuzz's user avatar
4 votes
1 answer
275 views

Should Feature Selection processes be apply on training data or on all data?

I've realized that on examples and guides, sometimes feature selection processes (correlation elimination, backward/stepwise) are applied on the train data after splitting all data but on the other ...
talatccan's user avatar
  • 173
4 votes
1 answer
2k views

Hive: How to calculate the Kendall coefficient of correlation of a pair of a numeric columns in the group?

In this wiki page there is a function corr() that calculates the Pearson coefficient of correlation, but my question is that: is there any function in Hive that ...
Marcin's user avatar
  • 235
4 votes
1 answer
451 views

How to make sure that the features learned by a neural network are not correlated?

Each layer of a neural network learns features of the input data. The first layer learns low-level features (e.g. edges in images). Each subsequent layer learns more abstract features. Then the ...
Vladislav Gladkikh's user avatar
4 votes
1 answer
704 views

Correlation between examples

I'm training a Neural Network for pattern recognition. I have a matrix of examples of size ($N$x$4$) with $N$ examples and $4$ variables. When I train the Network the Number of examples used for ...
juan9793's user avatar
4 votes
2 answers
6k views

Is $R^2$ an appropriate evaluation metric for k-Nearest Neighbors?

I found a source that stated that $R^2$ is the ”percentage of the response variable variation that is explained by a linear model.” (Source) Since kNN is not a linear model (it is nonparametric), is ...
covfefe's user avatar
  • 293
4 votes
1 answer
360 views

Minimize correlation between input and output of black box system

I am not sure if "minimize correlation" is the right title for this issue but I could not find a better sentence to describe what I would like to achieve. Let's say that I have a black box with ...
David's user avatar
  • 141
4 votes
2 answers
571 views

what is the difference in terms namely Correlation, correlated and collinearity?

A website says Correlation refers to an increase/decrease in a dependent variable with an increase/decrease in an independent variable. Collinearity refers to two or more independent variables acting ...
Subhash C. Davar's user avatar
4 votes
2 answers
568 views

Dealing with correlated features when calculating permutation importance

I have implemented the permutation importance calculation as found here in my attempt to identify features that contribute little to my model's (Gradient Boosted Tree model) predictive power. The ...
HFulcher's user avatar
  • 182
4 votes
2 answers
555 views

Machine learning algorithm that uses the Pearson or Spearman correlation?

I've come across linear and multiple regression, SVM, random forests. Does any know of a machine learning algorithm that uses the Pearson correlation or Spearman correlation? Best, Dave
Dave Nguyen's user avatar
4 votes
1 answer
1k views

SelectKBest and Correlation returns me excatly same feature selection. How?

Im working on selecting most effective features from a dataset with over that 2000 features. Im using different algorithms for that (selectKBest with chi-square, Extra Trees, Correlation etc.) But ...
justRandomLearner's user avatar
3 votes
2 answers
6k views

Can features negatively correlated with the target be used?

In feature selection (for a regression problem), can features that are negatively correlated with the target variable be chosen to predict the target? I don't think negative correlation means the ...
Bharathi's user avatar
  • 277
3 votes
2 answers
200 views

How can I determine the relationship between spam and weekdays?

I am trying to check if there is a correlation between spam emails and weekdays. My dataset looks like as follows: ...
LdM's user avatar
  • 165

1
2 3 4 5
9