Questions tagged [correlation]
A measure of the degree of linear association among a pair of variables.
397
questions
0
votes
2
answers
23
views
Correlation Analysis dealing with missing value, when the missing values are actually expected
So I am working on the correlation analysis in a dataset and trying to figure out the most sensible way to handle missing values. In my case, the missing values are expected.
To make it clearer, here ...
- 101
0
votes
0
answers
10
views
What is the best sampling strategy for correlation analysis?
I have a big dataset, and i want to finds subspaces with high correlation among features. I want to take only samples of data. So, what is the best sampling strategy in this context.
Thanks
- 21
0
votes
0
answers
17
views
Good candidate time series models for 70-100 monthly data points, also incorporate past and future exogenous covariates
Per title, we're trying to identify a good time series modeling technique for:
70-100 variables of monthly sales or volume data (2015 or 2018 to present)
Ability to forecast not only using trend and ...
- 101
0
votes
0
answers
28
views
Correlation in R between categorical and boolean variables
I want to use the data attached to see i there is correlation between a bootcamp that students attended and the job they end up getting. For example, does someone who attended a software engineering ...
- 1
0
votes
1
answer
101
views
Why does the order of columns in CCA change some results?
When running CCA from scikit learn, if the order is changed in the columns like (changing the order of the rows of both datasets together does not produce different results that I've seen -- not shown ...
- 101
0
votes
0
answers
31
views
What is reliability? How it is related to correlation coefficient?
We can compute real or population correlation(rho) by square-root of 1 minus R-squared.Is this a correct interpretation? Does population correlation mean a real correlation measured as square-root of ...
- 605
0
votes
0
answers
48
views
Need feedback on idea for new regularization term
I've been working on creating a regularization term that ensures that correlated attributes are given similar weights in a linear model. This helps to avoid some of the inconsistency in the weights of ...
- 3
0
votes
1
answer
63
views
Linear Model With Highly Correlated Attributes Producing Inconsistent Weights
I know that having correlated attributes violates the linear model assumption of independent attributes, and I'm not interested in creating a more sophisticated model to tease apart the dependent ...
- 3
0
votes
0
answers
12
views
ACF and PACF Autocorrelation
I have been trying to Understand correlation and tried it on my project and need help interpreting for a my case and in general.
I have the following results but finding it hard to interpret.
click ...
- 1
0
votes
0
answers
53
views
Correlation Matrix for Very Large Data Set
I am tasked to create a correlation matrix on a large portion of an MRI imaging data set (all numeric entries). The dimensions of the data is around 28,000x1,800. I have already set a correlation ...
- 101
0
votes
1
answer
36
views
Pearson coefficients different for same data?
Is it possible to have different Pearson coefficients over the same input data?
- 17
2
votes
2
answers
94
views
Edit friendly DDPM noise space
I was reading this paper, "An Edit Friendly DDPM Noise Space: Inversion and Manipulations". In page no. 4, they have mentioned that in DDPM, noise maps of consecutive steps are highly ...
- 130
0
votes
1
answer
41
views
Automatic statistical analysis / statistical "diff" tool?
I'm new to data science and wondered if there are ways to have a tool figure out relations between variables that may be relevant to a problem.
Imagine I have a log file that I have pre-processed to ...
0
votes
1
answer
78
views
How to find the correlation coefficient of one piece of data in a set?
I have a set of two pieces of data with a 0.85 coefficient. How can I calculate the coefficient for each individual set of data? For example, one set is 0-200 (x) and the other one is 0-$500,000 (y)....
1
vote
1
answer
50
views
How to proceed when correlation plot doesn't show as much multicorrelation as is seen by statsmodels variance_inflation_factor in a regression task?
I am working with the kaggle Blueberry Yield prediction dataset. There are 17 columns including the target variable. Below is the correlation heat map:
It can be seen that multiple features are ...
- 111
0
votes
0
answers
27
views
How can I correlate different types of variables?
first of all, my knowledge on this subject is very limited, therefore it may be a silly question.
I have 3 vectors, two of them are (distance vectors) in the range of 0 and 1, and the other is (...
- 1
0
votes
0
answers
47
views
What is degree assortativity coefficient of a complete undirected graph?
Because it computes the correlation coefficient of degrees and the correlation coefficient of constant arrays is not defined, networkx library returns ...
- 3
0
votes
2
answers
1k
views
corr() is giving an error. please help out of this problem and tell me what is this error about
when I am trying to run sns.heatmap(df.corr(),annot=True) this code in my jupyter notebook. this error is occuring. I cannot understand this problem. please help me.
0
votes
1
answer
53
views
Categorical feature relationship with target
When I'm working on dataset and want to explore different relationships between features and target, I often use visualization, only for regression with continous feature and continous target I use ...
- 51
0
votes
0
answers
18
views
Binary logistic regression vs generalized estimating equation (GEE) for time series
I have time series with 322 observations. My dataset contains financial data. My endogenous variable, "target" is a binary variable. My exogenous variables are two continuous variables: &...
2
votes
1
answer
336
views
If my target variable is binary, is it better to use Pearson's or Spearman's for my correlation vector?
I'm using a corr vector, combined with RFE, to perform feature selection.
I keep reading conflicting things online as to whether I should use Pearson's or Spearman's...
- 23
0
votes
0
answers
9
views
is it sound practice to normalise & sum numerical variables to create a single metric for a prediction model? e.g's provided
Does this question aim to understand if this is sound practice? if not, I would appreciate a suggestion.
The goal is to use this metric:
regression
classification
as a metric
profile_views
...
0
votes
0
answers
12
views
Two terms - observed correlation and population correlation are used frequently? Clarify the definitions
population correlation or true correlation is different from observed correlation? what is the purpose of finding population correlation?
- 605
0
votes
0
answers
17
views
Averaging subregion correlation coefficients into a single measure
I've got a 257x257 correlation matrix of functional connectivity (fMRI) data. It is a symmetric matrix where each value is the Pearsons correlation of the brain area in the row with the brain area in ...
- 101
0
votes
1
answer
44
views
Maximizing minimum correlation
What meaning has the weighted sum of a group of variables so that each weight is assigned to maximize the minimum resulting correlation of all these variables to the sum obtained?
- 1
0
votes
2
answers
181
views
How do we distinguish between correlated and un-correlated features/variables ? Is it relevant for a regression analysis?
Correlated and un-correlated terms are frequently used in data-science and understood as if they represent correlation coefficient. Is it the right way?
- 605
1
vote
1
answer
37
views
Best way to detect newly incoming anomalies in two timeseries?
I have two devices that both send data (let's say temperature). I need to be able to detect if one of the devices reports an unusual/anomalous reading. The case if both of the devices report a "...
- 13
0
votes
1
answer
154
views
Should highly correlated features be removed, even if they have different type of information?
A quick example for this: we have many feature and two of them are policy count and premium_total (for all policies). We are predicting the expected claim amount with GBM or RF. Both policy_count and ...
- 25
0
votes
1
answer
39
views
Best way to compare classification output between different locations
I ran a neural network for 20+ different locations across the United States. At each location I have a list of their predictions in an array. This looks something like this...
...
0
votes
0
answers
21
views
Time series analysis for document published date and read date
I want to find whether there is a particular hour when the author should publish the documents online to get maximum views/reads.
Data looks something like this -
Document ID
Published Date
Read Date
...
- 338
0
votes
0
answers
19
views
Check relationship between ordinal and categorical variable with four categories
how can we check relationship between an ordinal and categorical variable with 4 categories?
I have a variable with satisfaction score from 1-5, and other variable is distance from home like 1) <...
0
votes
1
answer
449
views
Shapley Values - How to interpret each value for each feature for a specific instance?
I am using Shap Values(the 'shap' module in python) to help me understand a bit better the relation between my features and my target. I am currently working on a binary classification problem.
I know ...
0
votes
1
answer
129
views
How can I take the cross-correlation between two discrete signal with slightly different discrete time points?
I have two discrete signals x1 and x2 with corresponding time points t1 and ...
- 1
0
votes
1
answer
73
views
How to correctly compute the correlation index of a column value from table in Python 3?
I have a data table of daily values for the past 2 years that looks like this, and I need to calculate the correlations between the data in Python. I have no background in data science, so I am afraid ...
0
votes
1
answer
48
views
Generate a set of values that has a given correlation with n given other sets of values
Given lists $L_1, \dots, L_n$ of, let's say, 2000 values each and arbitrary numbers $c_1, \dots, c_n$, is it possible to generate a random list of 2000 values that has correlation $c_i$ with $L_i$ for ...
0
votes
1
answer
1k
views
How does Pandas' Correlation Method Handle Non-Numeric Columns?
I'm using Pandas' .corr() method to figure out which columns I can eliminate from a large dataset. Some of those columns have non-numeric types.
How does Pandas ...
- 617
2
votes
1
answer
679
views
How do you Speed up the Calculation of a Correlation Matrix on a Large Dataset in Pandas?
I'm using a dataset with roughly 460,000 rows and 1,300 columns. I'd like to reduce the number of columns by seeing which have the largest effect on score using pandas' ...
- 617
2
votes
2
answers
423
views
What is the difference (interpretation) between the partial R^2 and the SHAP value for a linear regression model?
To calculate the coefficient of partial determination R2 for a given variable:
We calculate the R2 with and without that variable and substract them. This implies fitting a different model with and ...
- 185
0
votes
1
answer
51
views
What is the fastest way to detect lag and calculate cross correlation of two binary time series?
Example,
arr1 = array([0,0,0,1,1,1,1,0,0,0,0,0,1,1,0,0,0,0,0,0,1,0,1,1,1,1,1,0,0])
arr2 = array([1,1,1,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,1,1,1,1,1,0,0,0,0,0])
arr2 is almost perfectly correlated with ...
- 21
0
votes
0
answers
24
views
Which redundant feature should be use
I have two redundant features. A & B with 0.85 correlation. I know only one of them should be used to trained my model, but which feature should i use? A or B? Is there any method that can i use ...
- 138
2
votes
2
answers
475
views
Correlation vs Mutual Information vs let-the-model-decide
I recently encountered the Mutual Information concept, and started reading on it. As I saw that it can get non-linear relations, it seemed to me that it might be a more powerful method to choose which ...
- 25
0
votes
1
answer
21
views
Different Correlation Coefficents with different Time Ranges
I built a Time-Series that displays the price of the Electricty Price in South Italy and two of their most important commodities (commodities, gas) used to produce the eletrical energy. So I ordered ...
- 101
0
votes
1
answer
71
views
Is it necessary to have a perfect correlation when using linear regression?
I am working on predicting BMI against weight, using linear regression.
The scatter plot of the data can be found below.
As you can see in the plot, there seems to be low (or no) correlation between ...
0
votes
1
answer
66
views
R: the correlation is strong, but the graphic is very noisy
I have the following piece of code:
...
0
votes
0
answers
14
views
Find the most impactfuls parameters multivariate output unsupervised ML
I am currently on a proect where my df has more than 600 parameters of analog sensors (A parameters) and about 50 other parameters (F parameters). I want to find for each of these 50 parameters (F ...
- 1
1
vote
1
answer
178
views
Why Multicollinearity is a problem in machine learning algorithms
Is only a subset of algorithms are affected by the multicollinearity problem or all the machine learning algorithms?
What is the solution for this?
- 11
1
vote
1
answer
31
views
Remove Noise Caused by Other Variables to Predict an Expected Value
I have three variables measured at a sensor: Temperature (T), Humidity (H), and Methane Concentration (PPM). There are physical reasons why changes in T and H will influence PPM. I am interested in ...
- 113
0
votes
1
answer
53
views
Data preprocessing for Multiple Linear Regression Problem
For multiple linear regression problem, I have both categorical and numerical variables in the data. I am checking the correlation for numerical variables for EDA and standardizing them by taking log.
...
- 13
1
vote
0
answers
19
views
How to build a model with variables whose effects might be delayed?
As a personal project, I want to see how certain lifestyle choices (such as alcohol consumption, exercise time, hours worked/slept, time I get up and go to sleep) affect my mood. While I know the ...
1
vote
1
answer
105
views
Handling Covariate Shift and Multi-collinearity in same Dataset
The problem is related to Regression problem.
I am getting batches of data from a source of experiment which has approx 3k columns. However, I observed that almost 99% of the columns are highly ...
- 239