Questions tagged [correlation]

A measure of the degree of linear association among a pair of variables.

Filter by
Sorted by
Tagged with
0 votes
2 answers
23 views

Correlation Analysis dealing with missing value, when the missing values are actually expected

So I am working on the correlation analysis in a dataset and trying to figure out the most sensible way to handle missing values. In my case, the missing values are expected. To make it clearer, here ...
Sijan Bhattarai's user avatar
0 votes
0 answers
10 views

What is the best sampling strategy for correlation analysis?

I have a big dataset, and i want to finds subspaces with high correlation among features. I want to take only samples of data. So, what is the best sampling strategy in this context. Thanks
Imen F's user avatar
  • 21
0 votes
0 answers
17 views

Good candidate time series models for 70-100 monthly data points, also incorporate past and future exogenous covariates

Per title, we're trying to identify a good time series modeling technique for: 70-100 variables of monthly sales or volume data (2015 or 2018 to present) Ability to forecast not only using trend and ...
Vaslo's user avatar
  • 101
0 votes
0 answers
28 views

Correlation in R between categorical and boolean variables

I want to use the data attached to see i there is correlation between a bootcamp that students attended and the job they end up getting. For example, does someone who attended a software engineering ...
Max's user avatar
  • 1
0 votes
1 answer
101 views

Why does the order of columns in CCA change some results?

When running CCA from scikit learn, if the order is changed in the columns like (changing the order of the rows of both datasets together does not produce different results that I've seen -- not shown ...
user14094230's user avatar
0 votes
0 answers
31 views

What is reliability? How it is related to correlation coefficient?

We can compute real or population correlation(rho) by square-root of 1 minus R-squared.Is this a correct interpretation? Does population correlation mean a real correlation measured as square-root of ...
Subhash C. Davar's user avatar
0 votes
0 answers
48 views

Need feedback on idea for new regularization term

I've been working on creating a regularization term that ensures that correlated attributes are given similar weights in a linear model. This helps to avoid some of the inconsistency in the weights of ...
Brett L's user avatar
0 votes
1 answer
63 views

Linear Model With Highly Correlated Attributes Producing Inconsistent Weights

I know that having correlated attributes violates the linear model assumption of independent attributes, and I'm not interested in creating a more sophisticated model to tease apart the dependent ...
Brett L's user avatar
0 votes
0 answers
12 views

ACF and PACF Autocorrelation

I have been trying to Understand correlation and tried it on my project and need help interpreting for a my case and in general. I have the following results but finding it hard to interpret. click ...
Angie's user avatar
  • 1
0 votes
0 answers
53 views

Correlation Matrix for Very Large Data Set

I am tasked to create a correlation matrix on a large portion of an MRI imaging data set (all numeric entries). The dimensions of the data is around 28,000x1,800. I have already set a correlation ...
user123's user avatar
  • 101
0 votes
1 answer
36 views

Pearson coefficients different for same data?

Is it possible to have different Pearson coefficients over the same input data?
SSSOF's user avatar
  • 17
2 votes
2 answers
94 views

Edit friendly DDPM noise space

I was reading this paper, "An Edit Friendly DDPM Noise Space: Inversion and Manipulations". In page no. 4, they have mentioned that in DDPM, noise maps of consecutive steps are highly ...
shivani's user avatar
  • 130
0 votes
1 answer
41 views

Automatic statistical analysis / statistical "diff" tool?

I'm new to data science and wondered if there are ways to have a tool figure out relations between variables that may be relevant to a problem. Imagine I have a log file that I have pre-processed to ...
Evgeniy Berezovsky's user avatar
0 votes
1 answer
78 views

How to find the correlation coefficient of one piece of data in a set?

I have a set of two pieces of data with a 0.85 coefficient. How can I calculate the coefficient for each individual set of data? For example, one set is 0-200 (x) and the other one is 0-$500,000 (y)....
Dave Bell's user avatar
1 vote
1 answer
50 views

How to proceed when correlation plot doesn't show as much multicorrelation as is seen by statsmodels variance_inflation_factor in a regression task?

I am working with the kaggle Blueberry Yield prediction dataset. There are 17 columns including the target variable. Below is the correlation heat map: It can be seen that multiple features are ...
forest's user avatar
  • 111
0 votes
0 answers
27 views

How can I correlate different types of variables?

first of all, my knowledge on this subject is very limited, therefore it may be a silly question. I have 3 vectors, two of them are (distance vectors) in the range of 0 and 1, and the other is (...
Mustyby2's user avatar
0 votes
0 answers
47 views

What is degree assortativity coefficient of a complete undirected graph?

Because it computes the correlation coefficient of degrees and the correlation coefficient of constant arrays is not defined, networkx library returns ...
Neo's user avatar
  • 3
0 votes
2 answers
1k views

corr() is giving an error. please help out of this problem and tell me what is this error about

when I am trying to run sns.heatmap(df.corr(),annot=True) this code in my jupyter notebook. this error is occuring. I cannot understand this problem. please help me.
Subhajit Sarkar's user avatar
0 votes
1 answer
53 views

Categorical feature relationship with target

When I'm working on dataset and want to explore different relationships between features and target, I often use visualization, only for regression with continous feature and continous target I use ...
jxqbbb's user avatar
  • 51
0 votes
0 answers
18 views

Binary logistic regression vs generalized estimating equation (GEE) for time series

I have time series with 322 observations. My dataset contains financial data. My endogenous variable, "target" is a binary variable. My exogenous variables are two continuous variables: &...
Oskar Bieńko's user avatar
2 votes
1 answer
336 views

If my target variable is binary, is it better to use Pearson's or Spearman's for my correlation vector?

I'm using a corr vector, combined with RFE, to perform feature selection. I keep reading conflicting things online as to whether I should use Pearson's or Spearman's...
YoungboyVBA's user avatar
0 votes
0 answers
9 views

is it sound practice to normalise & sum numerical variables to create a single metric for a prediction model? e.g's provided

Does this question aim to understand if this is sound practice? if not, I would appreciate a suggestion. The goal is to use this metric: regression classification as a metric profile_views ...
Zaahir Ebrahim Dawood's user avatar
0 votes
0 answers
12 views

Two terms - observed correlation and population correlation are used frequently? Clarify the definitions

population correlation or true correlation is different from observed correlation? what is the purpose of finding population correlation?
Subhash C. Davar's user avatar
0 votes
0 answers
17 views

Averaging subregion correlation coefficients into a single measure

I've got a 257x257 correlation matrix of functional connectivity (fMRI) data. It is a symmetric matrix where each value is the Pearsons correlation of the brain area in the row with the brain area in ...
chainhomelow's user avatar
0 votes
1 answer
44 views

Maximizing minimum correlation

What meaning has the weighted sum of a group of variables so that each weight is assigned to maximize the minimum resulting correlation of all these variables to the sum obtained?
gabriel's user avatar
0 votes
2 answers
181 views

How do we distinguish between correlated and un-correlated features/variables ? Is it relevant for a regression analysis?

Correlated and un-correlated terms are frequently used in data-science and understood as if they represent correlation coefficient. Is it the right way?
Subhash C. Davar's user avatar
1 vote
1 answer
37 views

Best way to detect newly incoming anomalies in two timeseries?

I have two devices that both send data (let's say temperature). I need to be able to detect if one of the devices reports an unusual/anomalous reading. The case if both of the devices report a "...
Jamess11's user avatar
0 votes
1 answer
154 views

Should highly correlated features be removed, even if they have different type of information?

A quick example for this: we have many feature and two of them are policy count and premium_total (for all policies). We are predicting the expected claim amount with GBM or RF. Both policy_count and ...
morqueatsz's user avatar
0 votes
1 answer
39 views

Best way to compare classification output between different locations

I ran a neural network for 20+ different locations across the United States. At each location I have a list of their predictions in an array. This looks something like this... ...
Jack Cahill's user avatar
0 votes
0 answers
21 views

Time series analysis for document published date and read date

I want to find whether there is a particular hour when the author should publish the documents online to get maximum views/reads. Data looks something like this - Document ID Published Date Read Date ...
Kriti's user avatar
  • 338
0 votes
0 answers
19 views

Check relationship between ordinal and categorical variable with four categories

how can we check relationship between an ordinal and categorical variable with 4 categories? I have a variable with satisfaction score from 1-5, and other variable is distance from home like 1) <...
kusum shekhawat's user avatar
0 votes
1 answer
449 views

Shapley Values - How to interpret each value for each feature for a specific instance?

I am using Shap Values(the 'shap' module in python) to help me understand a bit better the relation between my features and my target. I am currently working on a binary classification problem. I know ...
Gabriel Monteiro's user avatar
0 votes
1 answer
129 views

How can I take the cross-correlation between two discrete signal with slightly different discrete time points?

I have two discrete signals x1 and x2 with corresponding time points t1 and ...
hokge's user avatar
  • 1
0 votes
1 answer
73 views

How to correctly compute the correlation index of a column value from table in Python 3?

I have a data table of daily values for the past 2 years that looks like this, and I need to calculate the correlations between the data in Python. I have no background in data science, so I am afraid ...
janjilecek's user avatar
0 votes
1 answer
48 views

Generate a set of values that has a given correlation with n given other sets of values

Given lists $L_1, \dots, L_n$ of, let's say, 2000 values each and arbitrary numbers $c_1, \dots, c_n$, is it possible to generate a random list of 2000 values that has correlation $c_i$ with $L_i$ for ...
akuea3728's user avatar
0 votes
1 answer
1k views

How does Pandas' Correlation Method Handle Non-Numeric Columns?

I'm using Pandas' .corr() method to figure out which columns I can eliminate from a large dataset. Some of those columns have non-numeric types. How does Pandas ...
Connor's user avatar
  • 617
2 votes
1 answer
679 views

How do you Speed up the Calculation of a Correlation Matrix on a Large Dataset in Pandas?

I'm using a dataset with roughly 460,000 rows and 1,300 columns. I'd like to reduce the number of columns by seeing which have the largest effect on score using pandas' ...
Connor's user avatar
  • 617
2 votes
2 answers
423 views

What is the difference (interpretation) between the partial R^2 and the SHAP value for a linear regression model?

To calculate the coefficient of partial determination R2 for a given variable: We calculate the R2 with and without that variable and substract them. This implies fitting a different model with and ...
skan's user avatar
  • 185
0 votes
1 answer
51 views

What is the fastest way to detect lag and calculate cross correlation of two binary time series?

Example, arr1 = array([0,0,0,1,1,1,1,0,0,0,0,0,1,1,0,0,0,0,0,0,1,0,1,1,1,1,1,0,0]) arr2 = array([1,1,1,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,1,1,1,1,1,0,0,0,0,0]) arr2 is almost perfectly correlated with ...
Imp's user avatar
  • 21
0 votes
0 answers
24 views

Which redundant feature should be use

I have two redundant features. A & B with 0.85 correlation. I know only one of them should be used to trained my model, but which feature should i use? A or B? Is there any method that can i use ...
Jovian Aditya's user avatar
2 votes
2 answers
475 views

Correlation vs Mutual Information vs let-the-model-decide

I recently encountered the Mutual Information concept, and started reading on it. As I saw that it can get non-linear relations, it seemed to me that it might be a more powerful method to choose which ...
Álvaro V.'s user avatar
0 votes
1 answer
21 views

Different Correlation Coefficents with different Time Ranges

I built a Time-Series that displays the price of the Electricty Price in South Italy and two of their most important commodities (commodities, gas) used to produce the eletrical energy. So I ordered ...
giacomomaraglino's user avatar
0 votes
1 answer
71 views

Is it necessary to have a perfect correlation when using linear regression?

I am working on predicting BMI against weight, using linear regression. The scatter plot of the data can be found below. As you can see in the plot, there seems to be low (or no) correlation between ...
tip. rock's user avatar
0 votes
1 answer
66 views

R: the correlation is strong, but the graphic is very noisy

I have the following piece of code: ...
njhkugk6i76g6gi6gi7g6's user avatar
0 votes
0 answers
14 views

Find the most impactfuls parameters multivariate output unsupervised ML

I am currently on a proect where my df has more than 600 parameters of analog sensors (A parameters) and about 50 other parameters (F parameters). I want to find for each of these 50 parameters (F ...
Art's user avatar
  • 1
1 vote
1 answer
178 views

Why Multicollinearity is a problem in machine learning algorithms

Is only a subset of algorithms are affected by the multicollinearity problem or all the machine learning algorithms? What is the solution for this?
Jagadeesh M's user avatar
1 vote
1 answer
31 views

Remove Noise Caused by Other Variables to Predict an Expected Value

I have three variables measured at a sensor: Temperature (T), Humidity (H), and Methane Concentration (PPM). There are physical reasons why changes in T and H will influence PPM. I am interested in ...
coomie's user avatar
  • 113
0 votes
1 answer
53 views

Data preprocessing for Multiple Linear Regression Problem

For multiple linear regression problem, I have both categorical and numerical variables in the data. I am checking the correlation for numerical variables for EDA and standardizing them by taking log. ...
Prasad Patil's user avatar
1 vote
0 answers
19 views

How to build a model with variables whose effects might be delayed?

As a personal project, I want to see how certain lifestyle choices (such as alcohol consumption, exercise time, hours worked/slept, time I get up and go to sleep) affect my mood. While I know the ...
Dave Moutardier's user avatar
1 vote
1 answer
105 views

Handling Covariate Shift and Multi-collinearity in same Dataset

The problem is related to Regression problem. I am getting batches of data from a source of experiment which has approx 3k columns. However, I observed that almost 99% of the columns are highly ...
Debadri Dutta's user avatar

1
2 3 4 5
8