Questions tagged [correlation]

A measure of the degree of linear association among a pair of variables.

Filter by
Sorted by
Tagged with
1
vote
1answer
14 views

How to identify recurring patterns in this set of transactional data

I'm working on a dataset of banking transactions and would like to find recurrent transactions. I've been mapping transactions per merchant in timeseries, and tried to use acf from statsmodels.tsa....
0
votes
0answers
7 views

How to compute Cramer's V if each variable 1 has more than 1 variable 2?

We're monitoring a plant's reactivity to nutrients. We administer different combination of nutrients to the plant every day and record it's reactivity on daily basis. Some of the nutrients/conditions ...
1
vote
0answers
21 views

Plotting heatmap seaborn dilemma

First i want to tell you that i'm trying to create a model in the right steps, which needs data exploration, data preprocessing (AFTER splitting the data into train/test), etc. For the part of data ...
0
votes
0answers
20 views

Titanic Dataset - Feature Engineering - Ticket feature

I am currently building my first machine learning model using the titanic dataset. After the data exploration, I decided to focus my attention on the 'Ticket' feature. One thing I have noticed about ...
1
vote
1answer
18 views

Which correlation test can be used on low count contigency table?

I need to find the correlation between the ratings given by two parties. Let's say we have 2 coaches, namely A and B. Coach A will rate the skills of the 12 players, either "Strong", "Moderate" or "...
1
vote
1answer
30 views

Pearson correlation coefficient - theory vs estimator

As far as I know when it comes to theory, we use Pearson correlation when we want to check linear correlation between two random variables, which are both continuous or discrete. For mixed case it's ...
1
vote
2answers
55 views

Which data set to use to find correlation between Predictor and response variables? Test data set? Training data set? or the entire data set?

I have a dataset with 4 predictor variables X1, X2, X3, X4, and one response variable Y. I have been asked to check the correlation between these variables and see how they are related and then use ...
2
votes
0answers
21 views

Which stage should the correlation analysis be done?

I was thinking about it, but I couldn't find a logical explanation. Mostly im following below steps after data become ready: Correlation analysis and elimination Apply dummy if categorical variables ...
1
vote
1answer
52 views

Is converting a categorical value into numerical needed to find a correlation?

I have a small dataset of 1300 observations x 20 features. They are all numerical but one, which is categorical; this was calculated independently and relates to each observation in any case. I'm now ...
1
vote
1answer
42 views

Correlation based Feature Selection vs Feature Engineering

I'm a bit confused about the superiority of Feature Selection over Feature Engineering or vice versa. Let's say I just want to get the best possible performance on a couple of models like a neural ...
0
votes
3answers
26 views

understading the corr heatmap

I want your help to analyze this correlation heat map if you look at this heat map what can you inference and what can you apply ?
1
vote
0answers
58 views

Effect of a single variable in a linear model

In a multiple linear regression (MLR) model, people normally look at the relationship between each predictor and the response variable to see if it is linear. However, in my example below, I show that ...
1
vote
1answer
22 views

how to see the correlation graph when there is lot of variables?

my variable has more then 4000 my corr() graph looks like this is there better way to look at this graph more clearly?
5
votes
3answers
490 views

What can be done with highly correlated variables (>.95 and <-.95)

I hope we can remove the highly correlated variables based on the feature importance may be with PCA etc. Is there anything we can do with highly correlated variables/ Thanks in advance !
2
votes
1answer
24 views

Several independent variables based on the same underlying data

I got a data containing, among others, two feature variables, which are based from the same underlying data (i.e. have mutual information), but they convey different information/message. How to handle ...
0
votes
0answers
47 views

Should the type of Boolean categorical features be numerical or categorical after encoding?

There are categorical features which have two different value in my dataframe next to numerical features. I've converted these categorical values to 0 or 1. I will apply correalation elimination on ...
3
votes
1answer
45 views

Should Feature Selection processes be apply on training data or on all data?

I've realized that on examples and guides, sometimes feature selection processes (correlation elimination, backward/stepwise) are applied on the train data after splitting all data but on the other ...
2
votes
0answers
23 views

Is there any solution for modified t-test in python for spatial data?

I am searching for a solution to conduct a modified t-test or any other possibility in order to compute significant or non-significant relations in a spatial dataset (which can be a stack of rasters ...
0
votes
0answers
18 views

Correlation between numeric variables of different meaning

I have a data set containing information about buildings, such as the area code, the neighborhood code, the number of floors, the year in which the house was built and the year in which restorations ...
2
votes
0answers
15 views

Correlation matrix on multi-time scales

I am trying to see how diversified a portfolio is, using the portfolio holdings weights and their return streams. For example, let's say that the portfolio has $3$ stocks, so I will have the $3x1$ W (...
0
votes
0answers
7 views

Dealing with built hierchical linear dependencies in my features

In a supervised learning problem, I am working on a dataset with a lot of hierachical linear dependencies between features. Let me be more clear about what I mean : In my dataset, there is a lot of ...
1
vote
0answers
45 views

Select the right distribution

I have a dataset like: ...
3
votes
2answers
48 views

Chose among highly correlated variables

I am working on a Kaggle dataset and I am trying to build a predictive model for the "Chance of Admit" (dependent variable) of students to the university of their interest. Below you can find the ...
2
votes
1answer
20 views

Clustering Weekday Weekend Data and Multicollinearity

Hi I have data of weekday and weekend step counts in which I extracted metrics from them such as the wd steps, we steps, standard deviation of wd steps, standard deviation of we steps and so on... <...
0
votes
0answers
5 views

What is the best approach (and why) to identify a conic section given the the points along its cross-section and its vector magnitudes over time?

I have an N-body simulation that calculates the position, velocity, and acceleration of each body at every frame over the course of some duration. As an example, I tested the algorithm using our solar ...
1
vote
1answer
16 views

Upper bound on 'relatedness'?

We have ~100 answers to a questionnaire with five questions (Q5). Independently from that, we have about 50, somewhat overlapping, features describing the people who answers the questions (F50). After ...
0
votes
1answer
192 views

Calibrating Correlation

I am facing a weird problem in my on going project and thought if someone here could help me out with this. Actually I have large data set. I have to perform a regression task on top of that. While ...
0
votes
0answers
18 views

How can I compare continuous and non-binary categorical data in Excel?

I'm trying to find the correlation between vehicle days on lot vs different categorical variables (color, make, model, etc). I know it's possible with binary data, but I haven't found a simple way to ...
3
votes
1answer
2k views

Pearson vs Spearman vs Kendall

What are the characteristics of three correlation coefficients and what are the comparisons of each correlation coefficients, Assumptions. Can somebody kindly take me through the concepts
1
vote
1answer
23 views

Relationship between two sets of dates

My wife has seizures a few times a month and takes medication for the seizures once a month. So I have two sets of dates - the dates of the seizures and the dates she got her medication. I'm trying ...
1
vote
3answers
87 views

Correlation between categorical variables based on the target distribution

Let $X$ be a category with very high cardinality and $Y$ be my target. when I look at $X$ distribution to $Y$ I see that some of the levels are very similar to each other . I would like to find a way ...
0
votes
1answer
58 views

Question mark on Correlation Matrix with RapidMiner

I'm using RapidMiner to evaluate correlation between attribute in my dataset. The problem is that a lot of values appear with '?'. Someone can help me? This is a sample of data
-1
votes
1answer
156 views

How to find and calculate correlation in a data set which has category and continuous variables? [duplicate]

I am working on an Insurance domain use case to predict if an existing customer will buy a second insurance policy or not. I have a few personal details saved under different categories like Marital ...
2
votes
1answer
229 views

Why is pandas corr() deleting columns?

I'm doing a basic correlation analysis but for some reason pandas corr() is deleting columns, not sure why. ...
2
votes
1answer
34 views

How do I assess whether two time series change together?

In this example, at timepoint 5, both signals move up together. I would like to quantify these similar movements, and ideally disregard the parts where the signals are almost constant. What ...
2
votes
2answers
79 views

What expresses if/how two variables are dependent on each other?

The accepted answer to question Why is a correlation matrix symmetric? includes this: The correlation matrix is a measure of linearity. It does not express how two variables are dependent on each ...
1
vote
1answer
40 views

List of Correlations to Correlation Matrix [closed]

I am using python to do some data analysis and I need to represent the following table as a correlation matrix. The correlation value is a value between -1 and 1. ...
0
votes
0answers
12 views

Generating correlated time series with different probability distributions

I have the following problem to solve: I need to generate two time series, each modeling a variable (lets call them VarA and VarB). For each of these variables, I have a PDF (PdfA, PdfB) which belongs ...
0
votes
0answers
45 views

How to analyzes seaborn's regplot of features from kaggle's houses prices and decide to remove a specific feature?

As a machine learning beginner, I am trying my best to, to solve kaggle's houses prices competition. I am doing pretty well. Some of the fields are confusing. The data set contains about 80 features,...
0
votes
2answers
32 views

Drawing validation set from test set

I am building a 3 neural network models on dataset that is already separated to train and test sets. From my analysis, I found that this dataset has values on test set which don't exist in the train ...
0
votes
1answer
30 views

Best way for correlation analysis

I am looking for an alternative way of analysing my data which will be used for ML. I am using Matlab to implement my code. Using my code, I obtain arrays which are used to describe signals. Each ...
2
votes
2answers
41 views

Monthly data versus yearly prediction

How would you see the relation of a monthly variable categorical with a yearly target value float for differents years in order to predict. In the farming industry the relation is not so simple for ...
0
votes
1answer
38 views

correlation between categorical(ordinal) and discrete(continuous) value [duplicate]

I am doing my bi variate analysis but right now looking to see the correlation between my atributes some are categorical 5 levels and others amount of money. how can I see the correlation between ...
1
vote
1answer
38 views

Checking if ML model is possible

How can I check if a machine learning model is feasible on a given dataset? What techniques like EDA, correlation etc. can be used to judge if a model is possible i.e. data and predictor variables ...
0
votes
2answers
18 views

How can I do the correlation between two estimators?

I'm working with several estimators of all kind. Then, I want to stack these estimators, and the best is if they have low correlation between them. I suppose that the correlation method depends on ...
1
vote
2answers
65 views

Finding correlation between MNIST digits

What way would be correct to calculate the correlation between say digit '1' and digit '7' images from MNIST? Taking average values of all digit '1' pixels and digit '7' pixels to compute correlation ...
0
votes
1answer
25 views

Pearson correlation make disapear the target column

I have a dataframe with some numerical and categorical values. I want to do some feature selection to visualise a low-dimensional split in the dataset when the target variable is the grade. Yet, when ...
0
votes
1answer
152 views

Correlation Matrix Issues: Can't Convert Variables to Numerics

QUESTIONS: Can you tell me how I can convert int and factor variables to numeric variables in R? Can you tell me why ...
9
votes
2answers
94 views

Features reduction for the not correlated data set

I am working with classification problem on a training data set, which have 100 features. All the features in pairs haven't visible correlation. One can see it in the example pair plot for the some of ...

1
2 3 4 5