Questions tagged [correlation]
A measure of the degree of linear association among a pair of variables.
223
questions
1
vote
1answer
14 views
How to identify recurring patterns in this set of transactional data
I'm working on a dataset of banking transactions and would like to find recurrent transactions.
I've been mapping transactions per merchant in timeseries, and tried to use acf from statsmodels.tsa....
0
votes
0answers
7 views
How to compute Cramer's V if each variable 1 has more than 1 variable 2?
We're monitoring a plant's reactivity to nutrients. We administer different combination of nutrients to the plant every day and record it's reactivity on daily basis.
Some of the nutrients/conditions ...
1
vote
0answers
21 views
Plotting heatmap seaborn dilemma
First i want to tell you that i'm trying to create a model in the right steps, which needs data exploration, data preprocessing (AFTER splitting the data into train/test), etc. For the part of data ...
0
votes
0answers
20 views
Titanic Dataset - Feature Engineering - Ticket feature
I am currently building my first machine learning model using the titanic dataset.
After the data exploration, I decided to focus my attention on the 'Ticket' feature.
One thing I have noticed about ...
1
vote
1answer
18 views
Which correlation test can be used on low count contigency table?
I need to find the correlation between the ratings given by two parties.
Let's say we have 2 coaches, namely A and B.
Coach A will rate the skills of the 12 players, either "Strong", "Moderate" or "...
1
vote
1answer
30 views
Pearson correlation coefficient - theory vs estimator
As far as I know when it comes to theory, we use Pearson correlation when we want to check linear correlation between two random variables, which are both continuous or discrete. For mixed case it's ...
1
vote
2answers
55 views
Which data set to use to find correlation between Predictor and response variables? Test data set? Training data set? or the entire data set?
I have a dataset with 4 predictor variables X1, X2, X3, X4, and one response variable Y. I have been asked to check the correlation between these variables and see how they are related and then use ...
2
votes
0answers
21 views
Which stage should the correlation analysis be done?
I was thinking about it, but I couldn't find a logical explanation.
Mostly im following below steps after data become ready:
Correlation analysis and elimination
Apply dummy if categorical variables ...
1
vote
1answer
52 views
Is converting a categorical value into numerical needed to find a correlation?
I have a small dataset of 1300 observations x 20 features. They are all numerical but one, which is categorical; this was calculated independently and relates to each observation in any case.
I'm now ...
1
vote
1answer
42 views
Correlation based Feature Selection vs Feature Engineering
I'm a bit confused about the superiority of Feature Selection over Feature Engineering or vice versa.
Let's say I just want to get the best possible performance on a couple of models like a neural ...
0
votes
3answers
26 views
understading the corr heatmap
I want your help to analyze this correlation heat map
if you look at this heat map what can you inference and what can you apply ?
1
vote
0answers
58 views
Effect of a single variable in a linear model
In a multiple linear regression (MLR) model, people normally look at the relationship between each predictor and the response variable to see if it is linear. However, in my example below, I show that ...
1
vote
1answer
22 views
how to see the correlation graph when there is lot of variables?
my variable has more then 4000
my corr() graph looks like this
is there better way to look at this graph more clearly?
5
votes
3answers
490 views
What can be done with highly correlated variables (>.95 and <-.95)
I hope we can remove the highly correlated variables based on the feature importance may be with PCA etc.
Is there anything we can do with highly correlated variables/
Thanks in advance !
2
votes
1answer
24 views
Several independent variables based on the same underlying data
I got a data containing, among others, two feature variables, which are based from the same underlying data (i.e. have mutual information), but they convey different information/message. How to handle ...
0
votes
0answers
47 views
Should the type of Boolean categorical features be numerical or categorical after encoding?
There are categorical features which have two different value in my dataframe next to numerical features. I've converted these categorical values to 0 or 1.
I will apply correalation elimination on ...
3
votes
1answer
45 views
Should Feature Selection processes be apply on training data or on all data?
I've realized that on examples and guides, sometimes feature selection processes (correlation elimination, backward/stepwise) are applied on the train data after splitting all data but on the other ...
2
votes
0answers
23 views
Is there any solution for modified t-test in python for spatial data?
I am searching for a solution to conduct a modified t-test or any other possibility in order to compute significant or non-significant relations in a spatial dataset (which can be a stack of rasters ...
0
votes
0answers
18 views
Correlation between numeric variables of different meaning
I have a data set containing information about buildings, such as the area code, the neighborhood code, the number of floors, the year in which the house was built and the year in which restorations ...
2
votes
0answers
15 views
Correlation matrix on multi-time scales
I am trying to see how diversified a portfolio is, using the portfolio holdings weights and their return streams.
For example, let's say that the portfolio has $3$ stocks, so I will have the $3x1$ W (...
0
votes
0answers
7 views
Dealing with built hierchical linear dependencies in my features
In a supervised learning problem, I am working on a dataset with a lot of hierachical linear dependencies between features. Let me be more clear about what I mean : In my dataset, there is a lot of ...
1
vote
0answers
45 views
3
votes
2answers
48 views
Chose among highly correlated variables
I am working on a Kaggle dataset and I am trying to build a predictive model for the "Chance of Admit" (dependent variable) of students to the university of their interest.
Below you can find the ...
2
votes
1answer
20 views
Clustering Weekday Weekend Data and Multicollinearity
Hi I have data of weekday and weekend step counts in which I extracted metrics from them such as the wd steps, we steps, standard deviation of wd steps, standard deviation of we steps and so on...
<...
1
vote
2answers
46 views
0
votes
0answers
5 views
What is the best approach (and why) to identify a conic section given the the points along its cross-section and its vector magnitudes over time?
I have an N-body simulation that calculates the position, velocity, and acceleration of each body at every frame over the course of some duration. As an example, I tested the algorithm using our solar ...
1
vote
1answer
16 views
Upper bound on 'relatedness'?
We have ~100 answers to a questionnaire with five questions (Q5). Independently from that, we have about 50, somewhat overlapping, features describing the people who answers the questions (F50). After ...
0
votes
1answer
192 views
Calibrating Correlation
I am facing a weird problem in my on going project and thought if someone here could help me out with this. Actually I have large data set. I have to perform a regression task on top of that. While ...
0
votes
0answers
18 views
How can I compare continuous and non-binary categorical data in Excel?
I'm trying to find the correlation between vehicle days on lot vs different categorical variables (color, make, model, etc). I know it's possible with binary data, but I haven't found a simple way to ...
3
votes
1answer
2k views
Pearson vs Spearman vs Kendall
What are the characteristics of three correlation coefficients and what are the comparisons of each correlation coefficients, Assumptions.
Can somebody kindly take me through the concepts
1
vote
1answer
23 views
Relationship between two sets of dates
My wife has seizures a few times a month and takes medication for the seizures once a month. So I have two sets of dates - the dates of the seizures and the dates she got her medication.
I'm trying ...
1
vote
3answers
87 views
Correlation between categorical variables based on the target distribution
Let $X$ be a category with very high cardinality and $Y$ be my target. when I look at $X$ distribution to $Y$ I see that some of the levels are very similar to each other . I would like to find a way ...
0
votes
1answer
58 views
Question mark on Correlation Matrix with RapidMiner
I'm using RapidMiner to evaluate correlation between attribute in my dataset. The problem is that a lot of values appear with '?'.
Someone can help me?
This is a sample of data
-1
votes
1answer
156 views
How to find and calculate correlation in a data set which has category and continuous variables? [duplicate]
I am working on an Insurance domain use case to predict if an existing customer will buy a second insurance policy or not. I have a few personal details saved under different categories like Marital ...
2
votes
1answer
229 views
Why is pandas corr() deleting columns?
I'm doing a basic correlation analysis but for some reason pandas corr() is deleting columns, not sure why.
...
2
votes
1answer
34 views
How do I assess whether two time series change together?
In this example, at timepoint 5, both signals move up together. I would like to quantify these similar movements, and ideally disregard the parts where the signals are almost constant.
What ...
2
votes
2answers
79 views
What expresses if/how two variables are dependent on each other?
The accepted answer to question Why is a correlation matrix symmetric? includes this:
The correlation matrix is a measure of linearity. It does not express how two variables are dependent on each ...
1
vote
1answer
40 views
List of Correlations to Correlation Matrix [closed]
I am using python to do some data analysis and I need to represent the following table as a correlation matrix. The correlation value is a value between -1 and 1.
...
0
votes
0answers
12 views
Generating correlated time series with different probability distributions
I have the following problem to solve: I need to generate two time series, each modeling a variable (lets call them VarA and VarB). For each of these variables, I have a PDF (PdfA, PdfB) which belongs ...
0
votes
0answers
45 views
How to analyzes seaborn's regplot of features from kaggle's houses prices and decide to remove a specific feature?
As a machine learning beginner, I am trying my best to, to solve kaggle's houses prices competition.
I am doing pretty well. Some of the fields are confusing.
The data set contains about 80 features,...
0
votes
2answers
32 views
Drawing validation set from test set
I am building a 3 neural network models on dataset that is already separated to train and test sets. From my analysis, I found that this dataset has values on test set which don't exist in the train ...
0
votes
1answer
30 views
Best way for correlation analysis
I am looking for an alternative way of analysing my data which will be used for ML. I am using Matlab to implement my code. Using my code, I obtain arrays which are used to describe signals. Each ...
2
votes
2answers
41 views
Monthly data versus yearly prediction
How would you see the relation of a monthly variable categorical with a yearly target value float for differents years in order to predict.
In the farming industry the relation is not so simple for ...
0
votes
1answer
38 views
correlation between categorical(ordinal) and discrete(continuous) value [duplicate]
I am doing my bi variate analysis but right now looking to see the correlation between my atributes
some are categorical 5 levels and others amount of money.
how can I see the correlation between ...
1
vote
1answer
38 views
Checking if ML model is possible
How can I check if a machine learning model is feasible on a given dataset? What techniques like EDA, correlation etc. can be used to judge if a model is possible i.e. data and predictor variables ...
0
votes
2answers
18 views
How can I do the correlation between two estimators?
I'm working with several estimators of all kind. Then, I want to stack these estimators, and the best is if they have low correlation between them.
I suppose that the correlation method depends on ...
1
vote
2answers
65 views
Finding correlation between MNIST digits
What way would be correct to calculate the correlation between say digit '1' and digit '7' images from MNIST? Taking average values of all digit '1' pixels and digit '7' pixels to compute correlation ...
0
votes
1answer
25 views
Pearson correlation make disapear the target column
I have a dataframe with some numerical and categorical values. I want to do some feature selection to visualise a low-dimensional split in the dataset when the target variable is the grade. Yet, when ...
0
votes
1answer
152 views
Correlation Matrix Issues: Can't Convert Variables to Numerics
QUESTIONS:
Can you tell me how I can convert int and factor variables to numeric variables in R?
Can you tell me why ...
9
votes
2answers
94 views
Features reduction for the not correlated data set
I am working with classification problem on a training data set, which have 100 features. All the features in pairs haven't visible correlation. One can see it in the example pair plot for the some of ...
